Scripting¶

This section will look at scripting. Scripting is used to perform complex or repetitive tasks without user intervention. All Linux commands which can be used on the commandline can also be used inside a script, including wild cards.

The most common reason for making a script is probably to avoid writing the same command again and again. On an HPC-sytem, such as the ones offered by NAISS, scripts are required to execute Linux commands and programs inside the batch system.

Starting with a motivational example¶

We start scripting with a simple example. The task at hand is to check how many lines of the file file.dat contain the string ‘ABCD’. This time we want to do this with a script.

#!/bin/bash
grep 'ABCD' file.dat > file_filtered.dat 
wc -l < file_filtered.dat > output.dat

$ chmod u+x my_first.sh

$ ./my_first.sh

What it does

• First line of the script: telling the system it should be executed in the bash shell, since commands differ between shells (the program loader is told to run the program /bin/bash as first argument). #! are called shebang. This has to be the first line.
• second line: search for the string ABCD in the file file.dat, then redirect the output to the file file_filtered.dat
• third line: run the command wc -l with the file file_filtered.dat as input. It then redirects the output to the file output.dat.

We can now execute two Linux commands with a single line.

Permissions¶

We have used the command chmod in the above example.

Permissions are needed to, among other things, make a file executable.

Let us take a look at an example

When running the command ls -la you should get an output similar to the below:

$ ls -la  
total 64
drwxr-xr-x 11 joachim  staff  352 Sep 29 16:47 .
drwxr-xr-x  6 joachim  staff  192 Sep 29 16:29 ..
-rwxr-xr-x  1 joachim  staff   99 Sep 29 17:30 analysis.sh
-rw-r--r--  1 joachim  staff  120 Sep 29 17:30 file.dat
-rw-r--r--  1 joachim  staff  128 Sep 29 17:30 file.txt
-rw-r--r--  1 joachim  staff  188 Sep 29 17:30 file2.dat
-rw-r--r--  1 joachim  staff   54 Sep 29 17:30 filtered_file.dat
drwxr-xr-x 13 joachim  staff  416 Sep 29 17:30 image
-rwxr-xr-x  1 joachim  staff  153 Sep 29 17:30 imagefind.sh
-rwxr--r--  1 joachim  staff   80 Sep 29 17:30 my_first.sh
-rwxr-xr-x  1 joachim  staff    9 Sep 29 17:30 program.sh

If you look at the left-most column, you see an example of permissions.

The permissions for a file, directory, or symbolic link has 10 “bits” and looks similar to this:

As shown, the first bit can be “-” (a file), “d” (a directory), or “l” (a link).

The following group of 3 bits are for the owner, then the next 3 for the group, and then the last 3 for all users. Each can have the r(ead), w(rite), and (e)x(ecute) permission set.

More scripting¶

Scripting is used to perform complex or repetitive tasks without user intervention. All Linux commands can be used in a script including wild cards.

We go back to our example script my_first.sh. There is a second file named file2.dat which also needs to be processed by the script. We could open an editor, change the contents of my_first.sh and re-run it. This is not really convenient. To create an improved script copy your script

$ cp my_first.sh count_ABCD.sh

#!/bin/bash
grep 'ABCD' $1 > file_filtered.dat 
wc -l < file_filtered.dat 

Check that count_ABCD.sh has executable permissions and executed the script as follows:

$ ./count_ABCD.sh file.dat

This should give the same result of 2 as before. When running the script count_ABCD.sh the $1 gets replaced with the first argument you write after the name of the script. The file file2.dat can now be processed without changing the script:

$ ./count_ABCD.sh file2.dat

We copy

$ cp count_ABCD.sh count_string.sh

#!/bin/bash
grep $2 $1 > file_filtered.dat 
wc -l < file_filtered.dat 

We can now search for ‘ABCD’ in both files:

$ ./count_string.sh file.dat ABCD
    2
$ ./count_string.sh file2.dat ABCD
    3

alternatively you can search for the word ‘line’ by changing the input at the prompt

$ ./count_string.sh file.dat line
    1
$ ./count_string.sh file2.dat line
    3

Commenting¶

It is useful to write comments into your script, to make their actions more understandable when e.g. you need to understand them in the future or want to share the with someone else.

Obviously this excludes the shebang in the first line of the script.

Open count_string.sh in an editor and add comments

#!/bin/bash
# return number of lines in a file containing a string
# usage: ./count_string file_name string

# search for lines with the string in the file
grep $2 $1 > file_filtered.dat 

# count the number of lines
wc -l < file_filtered.dat 

More advanced examples¶

#!/bin/bash
find $HOME/exercises/script/image -type f -name "*.png" -exec cp {} myimages \; 
find myimages -type f -name "*er*" > someimagesfiles.txt

#!/bin/bash
find . -type f -name "*.eps" -exec cp {} $HOME/figures \;
FIGFILES="$HOME/figures/*"
for f in $FIGFILES
do
    g=${f%.*}
    ps2pdf -DEPSCrop "$f" "$g.pdf"
done 

For more examples of (more useful) scripts, see for instance this list of 25 Easy Bash Script Examples.