Data Handling¶

This section will briefly cover compressing/decompressing files/directories, transferring files, and logging in.

Compressing and decompressing¶

Compressing files are done with utilities like gzip, bzip2, or zip.

gzip FILE

gunzip FILE.gz

Archiving¶

Archiving is generally done with tar.

A tarball is a commonly used name to refer to an archive file in the tar (Tape Archive) format.

A tarball can be compressed with something like gzip or bzip2.

tar [-options] <name of the tar archive> [files or directories which to add into archive]

Basic options:

    -c, --create — create a new archive;
    -a, --auto-compress — additionally compress the archive with a compressor which will be automatically determined by the file name extension of the archive. If the archive's name ends with *.tar.gz then use gzip, if *.tar.xz then use xz, *.tar.zst for Zstandard etc.;
    -r, --append — append files to the end of an archive;
    -x, --extract, --get — extract files from an archive;
    -f, --file — specify the archive's name;
    -t, --list — show a list of files and folders in the archive;
    -v, --verbose — show a list of processed files.

**Here follows some examples: **

tar -cvf DIRECTORY.tar DIRECTORY

tar -xvf DIRECTORY.tar

tar -zcvf DIRECTORY.tar.gz DIRECTORY

tar -zxvf DIRECTORY.tar.gz

More information can be found in HPC2N’s documentation’s Archiving and compressing section.

File transfer and syncing¶

There are several possible ways to transfer files and data to and from Linux systems: scp, sftp, rsync…

SCP¶

SCP (Secure CoPy) is a simple way of transferring files between two machines that use the SSH (Secure SHell) protocol.

$ scp sourcefilename user@hostname:somedir/destfilename

$ scp user@hostname:somedir/sourcefilename destfilename

SFTP¶

SFTP (SSH File Transfer Protocol or sometimes called Secure File Transfer Protocol) is a network protocol that provides file transfer over a reliable data stream.

enterprise-d [~]$ sftp user@kebnekaise.hpc2n.umu.se
Connecting to kebnekaise.hpc2n.umu.se...
user@kebnekaise.hpc2n.umu.se's password:
sftp> put file.c C/file.c
Uploading file.c to /home/u/user/C/file.c
file.c                          100%    1    0.0KB/s   00:00
sftp> put -P irf.png pic/
Uploading irf.png to /home/u/user/pic/irf.png
irf.png                         100% 2100    2.1KB/s   00:00
sftp>

sftp> get file2.c C/file2.c
Fetching /home/u/user/file2.c to C/file2.c
/home/u/user/file.txt  100%  1  0.1KB/s 00:00    
sftp> get -P file3.c C/
Fetching /home/u/user/file3.c to C/file3.c
/home/u/user/file.txt  100%  1  0.4KB/s 00:00    
sftp> exit
enterprise-d [~]$ 

rsync¶

rsync is a utility for efficiently transferring and synchronizing files between a computer and a storage drive and across networked computers by comparing the modification times and sizes of files.

rsync -rlpt username@remote_host:sourcedir/ /path/to/localdir

rsync -a /path/to/localdir/ username@remote_host:destination_directory

Much more information and examples can be found in the HPC2N documentation’s File transfer section.

Connecting with ssh¶

The ssh command is used for connecting to a remote computer.

Some useful examples:

ssh username@kebnekaise.hpc2n.umu.se

ssh -Y username@kebnekaise.hpc2n.umu.se

More advanced topics¶

This section will look at finding patterns (grep, awk, wild cards, regular expressions) and scripting.

Finding patterns¶

Here you will find descptions on how to search for files with specific patterns.

grep¶

This command searches for patterns in text files. FILE is the name of whatever file you want to look at. The file fil.txt is a good option here if you want to test.

grep 'word' FILE

grep -rine 'word' path/to/dir

awk¶

This command finds patterns in a file and can perform arithmetic/string operations.

awk '/snow/ {print$1}' FILE

Wild cards¶

Wild cards are useful ‘stand-ins’ for one or more character or number, that you can use for instance when finding patterns or when removing/listing all files of a certain type.

Wild cards are also called globbing patterns.

• ? represents a single character
• * represents a string of characters (0 or more)
• [ ] represents a range
• { } the terms are separated by commas and each term must be a wildcard or exact name
• [!] matches any character that is NOT listed between the [ and ]. This is a logical NOT.
• ** specifies an “escape” character, when using a subsequent special character.

myfile?.txt

r*d

r[a,i,o]ck

a[d-j]a

[0-9]

cp {*.dat,*.c,*.pdf} ~

rm thisfile[!8]*

Regular Expressions¶

Regular Expressions are a type of globbing patterns that are used when you are working with text.

Regular Expressions can be used with programs like grep, find and many others.

Some common examples of regular expressions:

• . matches any single character. Same as ? in standard wildcard expressions.
• \ is used as an “escape” character for a subsequent special character.
• .* is used to match any string, equivalent to * in standard wildcards.
• * the proceeding item is matched zero or more times. ie. n* will match n, nn, nnnn, nnnnnnn but not na or any other character.
• ^ means “the beginning of the line”. So “^a” means find a line starting with an “a”.
• $ means “the end of the line”. So “a$” means find a line ending with an “a”.
• [ ] specifies a range. Same as for normal wildcards. This is an ‘or’ relationship (you only need one to match).
• | This wildcard makes a logical OR relationship between wildcards. You can thus search something or something else. You may need to add a ‘' before this command to avoid the shell thinking you want a pipe.
• [^] This is the equivalent of [!] in standard wildcards, i.e. it is a logical “not” and will match anything not listed within the square brackets.

$ cat myfile | grep '^s.*n$'

Scripting¶

Scripting is used to perform complex or repetitive tasks without user intervention. All Linux commands can be used in a script including wild cards.

The most common reason for making a script is probably to avoid writing the same command again and again.

#!/bin/bash
grep 'ABCD' file.dat > file_filtered.dat
./program.sh < file_filtered.dat > output.dat

To change the permissions to execute a script (here named analysis.sh), for just the user, you could do:

$ chmod u+x analysis.sh

The above script can then be executed with

$ ./analysis.se

For more examples of (more useful) scripts, see for instance this list of 25 Easy Bash Script Examples.