November 19, 2016

Use Oracle Data Base in Command Line


Cadmus Asks the Delphic Oracle Where He Can Find his Sister, Europa by Hendrik Goltzius, 1615


Sometimes I need to make some quick changes in data base. Or I need to do several changes using some patterns and templates. In that case I don't want to open editor (I use DataGrip) and wait for loading all resources, connecting to DB, opening SQL window and inserting or copying SQL query. I am that kind of person who keeps open only those applications that I am going to use in next 30 min. If I don't plan to use something in next half an hour I close it, so every opening takes some time and effort.

The best way to save time and effort is command line or, actually, scripts. SQLPlus is a great tool for using Oracle data base in the console. You can find instructions about installing SQLPlus on MacOS on StackOverflow: Oracle Sqlplus client on Mac.

The annoying thing about using data base in the terminal is connecting: you have to remember and write all credentials of data base every time you want to connect to it. But, as usually, scripting resolves this problem. I wrote simple Bash script connect-to-oracle-db.sh, that opens connection using small alias as an input. For example, I write sql test in terminal and it opens connection with test data base where I can write some SQL right ahead. Instead of test it can be any base that I work with: demo, live, production etc.

Second script that I use is changing some value in specific data base in specific table. Some times one of the clients asks me to change specific data in their DB, so now instead of opening an editor I can just run a script with parameter given by client and it changes all the data, which saves me time and doesn't disturb my attention on other tasks so much. The script looks something like that:

#!/bin/bash

parameter=$1
sqlplus= # Path to SQLPlus
username= # DB username
password= # DB password
db= # DB name

SQL="
  UPDATE some_table
    SET some_column = some_value
    WHERE some_other_column = '$parameter';"

"$sqlplus" "$username/$password @$db" << HERE

$SQL
commit;
quit
HERE

Of course, SQLPlus doesn't replace editors like SQL Developer or DataGrip, but it can save a lot of time and effort in performing small and routine tasks.

October 24, 2016

Why I Don't Like Testing Conferences


The painting: "The Witches’ Cove" by Jan Mandijn

The best book about software testing has following introduction: "This book is about software development as we've experienced it." ("Lessons Learned in Software Testing" by Cem Kaner, James Bach, Bret Pettichord). Because you can't talk about testing without the context of the general development itself.


I like conferences – they are usually very inspiring, motivating and sometimes challenging. Visiting testing conferences gives a lot of ideas how to do my job better, but almost always that means improving some processes at the project. And it's almost impossible to change some steady process if more than 10 (or even 5) people are involved in it. To change the process you need to convince all team members that it brings benefits to the project or product. And to convince team members you need to retell the story you heard on the conference (which usually is as long as the conference talk or even longer with all the preparations you need to do) and to be talented speaker (usually inspiring speakers at the conference are good at speach), which in major cases is not true. So, wouldn't it be better that all (or maybe the key ones) team members just visited the conference all together to hear the same talk from experienced speaker and be inspired all at a time?

Majority of the people in the audience is having some ideas during the talk, they are very inspired and willing to do some changes in their project, they came back to work and start to talk about these changes with developers or project managers and then... they get couple of arguments why it won't work in this specific project. And this person, who visited the conference, is not so skillfull as the speaker to pitch other team members. So everyone thinks he is a boring tester who constantly offers some silly ideas.

This is not just impractical, this is harmfull. It brings discord between programmers and testers (and analytics, but programmers vs tester is the most popular confrontation). Programmers doesn't understand why testers suggest to do their life harder, because they haven't heard the same speach. For example, the idea to involve testers into the development as early as possible may seem to be silly if you hear that from one junior tester who visited some conference ("he doesn't understand anything at the early stage and I don't have time to explain it" – may say some programmer). But the same idea from the experienced speaker on the stage is not so silly anymore (at least you need to have a proper argument to argue with it).

I'd like to have conferences about software development generally, where all roles can participate. Surely, there should be specific conferences for testers and programmers, where speakers may talk about how to automate tests, which tools can be used, how to do security or performance testing. However questions like why we need automation, why we need security, at what stage of the project we need to thing about security, how ofter we need to release updates in production should be convered in general conferences, because these are the problems where all team members are involved. The problem is that I don't know any widely spread good conference about software development generally - all good conferences are role specific.

After all, all team members have one common goal – to create a product (good teams have goal to create a qualitative product). Both testers and developers works for the same goal, but visiting different conferences they start to see the same goal from two different perspectives.

September 7, 2016

Checking Deployments on Tomcat Server Without Web Manager

In some cases Tomcat web manager is disabled (for example, in production for security reasons). Then the only way to see deployments and their statuses is to use Tomcat API. To list deployed applications you may do following request:
http://localhost:8080/manager/text/list
And you will get something like this:
The problem is that if you have a lot of applications such output is not very easy to read – you can't say is there any stopped applications without reading all rows. For that case you can use awk to color the output, which is much more informative:
The next problem is that awk command is quite long and you don't want to type something like this every time:
curl http://localhost:8080/manager/text/list | sort | grep ^/ | awk '{ gsub("running", "\033[32m&\033[0m"); gsub("stopped", "\033[31m&\033[0m"); gsub("\\:[0-9]+", "\033[34m&\033[0m"); gsub("^/.+:", "\033[36m&\033[0m"); gsub("[0-9]+$", "\033[33m&\033[0m"); print }'
So you can use a script, that returns you colorful list of deployments without typing any URLs and regexps:
./show-status-of-applications-tomcat.sh

There is one other case, when script is even more convinient than web manager – if your environment is running on many servers and clusters (like production, again). In that case script can show information about all clusters and servers on one page:
I have separate script show-status-of-prod-applications-tomcat.sh for that, but it's possible to merge them into one file or modify it to work with parameters (for example, give a manager URL as a parameter).

All scripts are available in my GitHub repo github.com/iriiiina/scripts and don't forget about aliases!

August 9, 2016

Probe Testing in The Martian

The story from The Martian about testing the probe is worth a post (some scenes and phrases are skiped).








July 8, 2016

Bachelor Thesis "Version Update Automation Using Scripting Language Bash"


I've finally finished University of Tartu, so I am a Bachelor of Science in Engineering now!

My work was about version updater bash script, that I made on my work for updating Java applications on different web servers. I once wrote a post about first version of this script – Scripting For Automated Update (Tomcat 6) [DEPRECATED], – the final version has more features and much more code.

The thesis is written in Estonian and can be found in GitHub – Version Update Automation Using Scripting Language Bash.pdf (or in UT registry).

The script itself is open-source (currently only Tomcat 8 part) and can be used in other projects. It's located in GitHub repo – github.com/iriiiina/version-updater, the manual about install, configuration and usage is in GitBook – iriiiina.gitbooks.io/version-updater-manual.

And here are slides of defence (also in Estonian). I don't know why you may need them, but since we are talking about thesis I'll leave them here:

I feel proud that I got A for the thesis, but I don't have any good feeleings about finishing the university itself. Two years ago I wrote a post Secrets of a Buccaneer-Schoolar by James Marcus Bach where I explained my opinion about the university and hight schools, so it wasn't the priority for me. I decided to finish just to not loose already gotten points for finished courses – I've already passed 99% of the programm, so it would be a shame to spend that time for nothing.

June 17, 2016

Bash Scripts for Transfering Files Between Server and Local Computer


Yes, Bash scripts again! Two scripts for copying files from server to local computer and vice versa. Usually I use them if I want to work with some text files in graphical editors, not in Vim. So I can download the file to my local computer and then upload it back to the server.

./copy-file-from-server.sh [SERVER] [FILE]
./copy-file-to-server.sh [SERVER] [FILE]

Basically, it's the same as using scp command, but scripts allow you to define short names for servers. For example, instead of username@192.168.1.1, username@some.hostname.com or something1-23-45-678-9.eu-east-0.compute.amazonaws.com you can use short names, like test, demo, latest etc.

To set your own hosts in the script you need to modify function setHost(). In the copy-file-from-server.sh script you can also set a fixed directory where you want to download files. Currently it downloads to the same directory where script was run, but you can change it in the downloadFile() function.

Also, don't forget about aliases! You can set some short name to the script and use it in every directory.

June 7, 2016

Script for Sending E-mails About Certain JIRA Issues


"Still life with various Unix shells" by Bartholomeus van der Ast (source)


Recently I wrote a new Bash script, that can be useful for others.

./notify-about-issues.sh
It sends e-mail when some specific JIRA JQL query returns any result. You can create any JQL query you want: return issues that were created in specific project, return issues that were commented by some specific user, return issues where status was changed etc. Probably you want to limit the query by period to monitor only recent changes. This period can be the same as the frequency of the e-mail sending. For example, if you want to check this filter every 10 minutes, then you can check changes in last 10 minutes.

To implement this script you need to define following variables in the code:
$jiraUrl – URL of your JIRA
$filter – JQL query, encoded for URL (you can use some online URL encouder/decoder)
$user – JIRA username
$password – JIRA password (authentication is very primitive and not safe, but you can use some general credentials or keep the script file in safe place)
$to – e-mail where notification should be sent
$from – e-mail from whom notification should be sent (could be the same as to)
$subject – subject of the e-mail

Automatic running can be configured with crontab – read post 10 Mac OS X Terminal Commands for Testers for details.

The most complex part is to write a propper JQL query that returns exactly what you want. You can use Advanced searching documentation for complex queries or ask an advice in Atlassian Answers.

May 27, 2016

Web Application Security Training by Clarified Security


You shouldn't find any EXIF data in this picture.

I've participated in the Web Application Security Training by Elar Lang from Clarified Security. It's very detailed and thorough hands-on 4 days training for web programmers, testers and all people, that are related to web development.

As for me, it's must-do for all web developers and testers. If you look at the content (especially client-side part), then it may seem to be too general and basic, but actually these well known things are explained in a quite deep and advanced level.

Organization is really nice feature of this training: first we had 2 days of client-side part and next week 2 days of server-side part, so between them I had 3 working days to work through client-side attacks on my application. Also, there can be maximum 12 participants, which makes the treaining more individual.

In summary – I really recommend it to people, who thinks that their software is safe or that there is nothing to wory about even if it's not.

October 23, 2015

10 Mac OS X Terminal Commands for Testers


I really like terminals, command lines, bash scripting etc. They are handy and easy way to automate some routine tasks in my everyday work. Thats why after I saw article Eight Terminal Utilities Every OS X Command Line User Should Know I decided to write something like this, but for testers.

These are OS X commands, but some of them are also working on Linux systems and Windows (if use some UNIX-like terminal like Babun, CygWin or Cmder).

This is defently not the complete list. Here you can find the complete list of all OS X commands, but this post is about my favourite and frequently used (almost) commands.

1. alias
It's the most useful command of all – it allows to replace some command with a short word. That means, that if you have some long command that you regularly run in the terminal you can assign alias to it and use some simple short substitution. For example, you always monitor application logs using tail -1000f application.log, but instead of writing this whole string every day you can create alias like logs and write only 4 letters into the command line. Moreover, you can use autocomplete with aliases (TAB key), just like with regular commands. You can see my previous post Bash Scripts for Working With Documentation for more examples about using alias.

There is one trap that many beginners fall into – you can just type following command into your command line:
alias logs='tail -1000f application.log'
but in that case you create alias only for your current session. That means it'll disappear after you close it. To create permanent aliases you need to put them into specific file, which is usually ~/.bash_aliases. You may also need to add following code into your ~/.bash_profile file (you can add it just to the end):
# Aliases
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
And you need to restart your session for taking these changes into use (just close-open you terminal or exit-login to server).

2. crontab
This one allows you to automate some commands. For example, you have some documentation on SVN (or some similar version control system) and you need to update it regulary to get the latest state. You can add following job into your cron schedule, which updates SVN directory every 10 minutes:
*/10 * * * * svn update ~/Documents/SVN-documentaion
To open crontab editor, where you can insert previous command, you need to run:
crontab -e
(or sudo crontab -e if you are not under admin user)
See Ubuntu CronHowTo article for understanding what are these asterisks mean.

3. grep
Grep is a very powerful tool for finding strings in any output (log files, query responses, script output etc), that you can perform. I use it to filter log files. For example, I can get all rows, where ERROR word is present:
tail -1000 application.log | grep ERROR
Or I can get all rows, where todays date is mentioned in the beginning of the row:
tail -1000 application.log | grep ^23.10.2015
Grep works with regular expressions, so you may need to understand them to perform some complex filtering, but simple search by one word may be enough for you.

4. find
Find searches files in directory. Again, in my previous post Bash Scripts for Working With Documentation I wrote about scripts for finding documents by name or by content – the whole funtionality is done by find there. So I use it in by daily work for finding documents with required data. You can find (ha-ha) a lot of examples in the following article: Find Command in Unix and Linux Examples.

5. open
Simply opens a file, a folder, an application or URL, just as if you had double-clicked the file's icon. It's useful if you first have done some complicated search of required file (using find), got the path of it and open it (with default or some other application).
Second useful case – opening directories (especially hidden ones). For example, open . opens current directory.

6. awk
AWK is actually programming language designed for text processing. It can be used as reporting tool and I usually use it to beautify my scripts' output – for example, to color some words. I am not sure that it's useful for everyday life, but if you want to write your own script, it better has meaningful output.

7. scp
It stands for Secure Copy. It is useful for somebody who works with servers (application updating, logs monitoring etc). With this command you can copy files from one server to other, or from server to your computer and vice versa. You can find some examples in Example syntax for Secure Copy (scp), I copy here one of them:
scp your_username@remotehost.edu:foobar.txt /some/local/directory

8. pbcopy / pbpaste
These two commands may save you from endless scrolling: pbcopy copies output to the clipboard, pbpaste inserts content from the clipboard. For example, you have some log file, that you want to investigate on your computer in you favorite editor. The simplest way is to copy its content to clipboard and paste it into another document. With these commands you don't need to scroll anything to select the text, or google "how to select the whole text in vi" or anything like this, just type:
cat application.log | pbcopy

9. screencapture
Honestly, I don't use this command in my daily work, but I have a feeling that it's important. It does what it says – captures the screenshot. You can set time delay to it and you can write a script for time lapse (taking screenshot every 10 seconds or something like this). It just happens, that I use screenshots quite rarely in my work, but I guess it can be interesting command for testers, who use them a lot.
You can see examples here: Take a screen capture from the command line.

10. sips
Last (and least) one is SIPS – Scriptable Image Processing System. You can use it for processing files, for example, resize all PNG files in current directory to 1024×768 (ignoring aspect ratio):
sips -z 768 1024 *.png
Again, I don't work with screenshots and picture files a lot, but it's the simplest and the fastest way (and cheapest one, comparing to some Photoshop) to process a large amount of picture files (or if you want to work with pictures in your bash script).



Don't forget the alias! You don't need to know all these commands by heart – you can just figure out how you can use them and which parameters suits for your case, put it into the alias thereafter use only short and nice commands.

BONUS!
telnet towel.blinkenlights.nl
This one is just for fun (once you've already opened the terminal). Some guys did pretty awsome ASCII version of Star Wars IV. You can run it over the Telnet. I don't know how long does it last – I've watched 15 minutes and Luke only meets Chubaka and Han Solo, so I guess they did the whole movie.

September 30, 2015

Bash Scripts for Working With Documentation

Recently I wrote a couple of bash scripts for working with specification documents. They solve sort of general problems, so I decided to share them here.

All scripts can be downloaded from my GitHub repo: github.com/iriiiina/scripts

All scripts can be executed in UNIX terminal (for Windows users there are alternatives such as Babun, Cygwin, Cmder). I also recommend to use aliases: just pick up short and unique command, specify the path to the script and you can excecute this script with short command from wherever you are (like spec bug.01.1 for finding specification with such text in the title).


find-specification-by-name.sh
./find-specification-by-name.sh [text_in_file_title]
Script looks for .doc files (extension can be changed) in a specific directory (which can be also changed) where title contains text, that user gives to the script.
In the example I have directory specifications with 4 documents, but only 3 of them contain word bugs in the title. There is also directory Archive with old documents, which script also finds, but shows Archive folder in red, so user can easily differ old documents from new ones (you can also change words that should be colored).

Rows, that you may want to change to use this script in your way:
  • 6 path="/Users/irina/Desktop/specs" – here you should specify the absolute path (not relative, because you want to execute this script from different places) of the directory, where you have all your specifications (usually it's CVS of SVN root directory).
  • 31 find $path -iname "*$text*" | awk -v text="$text" -v lower="$lower" -v upper="$upper" '{ gsub(text, "\033[36m&\033[0m"); gsub(lower, "\033[36m&\033[0m"); gsub(upper, "\033[36m&\033[0m"); gsub("Arhiiv", "\033[31m&\033[0m"); print }' – the awk part is responsible for coloring. Codes like \033[36m are colors – particualary this one means cyan, but you can change it. Also you need to change word "Arhiiv", which helps to differ old documents. Or you can remove this part, if you don't want to color anything.

open-last-specification.sh
./open-last-specification.sh [text_in_file_title]
This one is doing basically the same as the previous one, except that it automatically opens lastly modified file. You can use it, if your components have unique identifiers and specs about these components have many versions, like this:
Rows that you may want to change:
  • 6 path="/Users/irina/Desktop/specs" – same as in the previous one, absolute path to directory with your specs.
  • 43 function checkIsFileIsDoc(){} – in my case I want to open only .doc files – I don't want to open folders or excel files, but you may want to remove this check.

find-text-in-specification.sh
./find-text-in-specification.sh [text_in_the_file_content]
This one is also very similar to first one, only it finds files with given text in the content.
For example, I want to know in which specification is mentioned data base column BUG.STATUS:
Rows that you may need to change are the same as in the first one.

nortal-logo.sh
I always wanted to print some pictures with bash script. So here are my first experiences.
Logo of company where I work. It was picked as easy to reproduce, because it has strict lines.

bug.sh