Navigating the File System¶

This section is going to be a high-level talk about the Linux filesystem concepts, not a low-level description of filesystem types.

The Linux filesystem directory structure starts with the top root directory, which is shown as /. Below this are several other standard directories. Of particular interest are usr/bin, home, usr/lib, and usr/lib64. A common directory which you will also often find is usr/local/bin.

The picture on the right shows typical subdirectories under / (note that the command ‘tree’ does not work at all HPC centres, though it does work on Tetralith - see the page tree under “Extras section for how to install if it is missing). Some of the directories have a symbolic link to a different name - this is often done to make it quicker to write, but can also be for compatibility reasons since some software have hardcoded paths.

• usr/bin: contains (most) of the system-specific binaries
• usr/local/bin: non-system binaries. often locally compiled/maintained packages
• home: where the home directories of the users of the system are located
• usr/lib: kernel modules and shared library images needed to boot the system and run commands in the root filesystem
• usr/lib64: same as /lib, just for 64-bit libraries

User-installed binaries are often located in /opt.

ls - listing files/directories¶

The ls command is used to list files. If you just give the command “ls” with no flags it will list all files in the current directory except for hidden files.

ls [flags] [directory]

This way you can to list files/subdirectories for any directory, but the default one is the one you are currently standing in.

Some examples:

• ls / lists contents of the root directory
• ls .. lists the contents of the parent directory of the current
• ls ~ lists the contents of your user home directory
• ls * lists contents of current directory and subdirectories

Commonly used flags:

• -d */ lists only directories
• -a lists content including hidden files and directories
• -l lists content in long table format (permissions, owners, size in bytes, modification date/time, file/directory name)
• -lh adds an extra column to above representing size of each file/directory
• -t lists content sorted by last modified date in descending order
• -tr lists content sorted by last modified date in ascending order
• -R lists contents recursively in all subdirectories
• -s list files with their sizes
• -S sort files/directories by size in descending order
• -Sr sort files/directories by size in ascending order

To get more flags, type ls --help or man ls in the terminal to see the manual.

[x_birbr@tetralith1 mytestdir]$ ls
myfile.txt  myotherfile.txt  testdir1  testdir2

[x_birbr@tetralith1 mytestdir]$ ls -a
./  ../  myfile.txt  myotherfile.dat  testdir1/  testdir2/

[x_birbr@tetralith1 mytestdir]$ ls -l
total 3
-rw-rw-r-- 1 x_birbr x_birbr   27 Sep 11 11:43 myfile.txt
-rw-rw-r-- 1 x_birbr x_birbr   33 Sep 11 11:43 myotherfile.txt
drwxrwxr-x 2 x_birbr x_birbr 4096 Sep 11 11:40 testdir1
drwxrwxr-x 2 x_birbr x_birbr 4096 Sep 11 11:39 testdir2

[x_birbr@tetralith1 mytestdir]$ ls -la
total 5
drwxrwxr-x 4 x_birbr x_birbr 4096 Sep 11 11:43 .
drwx------ 3 x_birbr x_birbr 4096 Sep 11 11:43 ..
-rw-rw-r-- 1 x_birbr x_birbr   27 Sep 11 11:43 myfile.txt
-rw-rw-r-- 1 x_birbr x_birbr   33 Sep 11 11:43 myotherfile.txt
drwxrwxr-x 2 x_birbr x_birbr 4096 Sep 11 11:40 testdir1
drwxrwxr-x 2 x_birbr x_birbr 4096 Sep 11 11:39 testdir2

[x_birbr@tetralith1 mytestdir]$ ls -lah
total 5.0K
drwxrwxr-x 4 x_birbr x_birbr 4.0K Sep 11 11:43 .
drwx------ 3 x_birbr x_birbr 4.0K Sep 11 11:43 ..
-rw-rw-r-- 1 x_birbr x_birbr   27 Sep 11 11:43 myfile.txt
-rw-rw-r-- 1 x_birbr x_birbr   33 Sep 11 11:43 myotherfile.txt
drwxrwxr-x 2 x_birbr x_birbr 4.0K Sep 11 11:40 testdir1
drwxrwxr-x 2 x_birbr x_birbr 4.0K Sep 11 11:39 testdir2

[x_birbr@tetralith1 mytestdir]$ ls -latr
total 5
drwxrwxr-x 2 x_birbr x_birbr 4096 Sep 11 11:39 testdir2
drwxrwxr-x 2 x_birbr x_birbr 4096 Sep 11 11:40 testdir1
-rw-rw-r-- 1 x_birbr x_birbr   27 Sep 11 11:43 myfile.txt
-rw-rw-r-- 1 x_birbr x_birbr   33 Sep 11 11:43 myotherfile.txt
drwx------ 3 x_birbr x_birbr 4096 Sep 11 11:43 ..
drwxrwxr-x 4 x_birbr x_birbr 4096 Sep 11 11:43 .

[x_birbr@tetralith1 mytestdir]$ ls *
myfile.txt  myotherfile.dat

testdir1:
file1.txt  file2.sh  file3.c  file4.dat

testdir2:
file1.txt  file2.txt  file3.c

[x_birbr@tetralith1 mytestdir]$ cd testdir1
b-an01 [~/mytestdir/testdir1]$ ls -l
total 2
-rw-rw-r-- 1 x_birbr x_birbr 31 Sep 11 11:47 file1.txt
-rw-rw-r-- 1 x_birbr x_birbr 16 Sep 11 11:49 file2.sh
-rw-rw-r-- 1 x_birbr x_birbr 74 Sep 11 11:49 file3.c
-rw-rw-r-- 1 x_birbr x_birbr 25 Sep 11 11:50 file4.dat

[x_birbr@tetralith1 mytestdir]$ ls -ls
total 2
1 -rw-rw-r-- 1 x_birbr x_birbr 31 Sep 11 11:47 file1.txt
1 -rw-rw-r-- 1 x_birbr x_birbr 16 Sep 11 11:49 file2.sh
1 -rw-rw-r-- 1 x_birbr x_birbr 74 Sep 11 11:49 file3.c
1 -rw-rw-r-- 1 x_birbr x_birbr 25 Sep 11 11:50 file4.dat

chmod - change permissions¶

The command chmod is used to change permissions for files and directories.

The permissions for a file, directory, or symbolic link has 10 “bits” and looks similar to this:

As shown, the first bit can be “-” (a file), “d” (a directory), or “l” (a link).

The following group of 3 bits are for the owner, then the next 3 for the group, and then the last 3 for all users. Each can have the r(ead), w(rite), and (e)x(ecute) permission set.

chown - change ownership¶

To change ownership of a file or directory, use the command chown.

chown [OPTIONS] USER[:GROUP] FILE(s)

Create and remove directories/files¶

This section contains a few examples of how to work with files and directories through command line interface.

• mkdir DIR: Create a directory DIR
• rm -rf DIR: Remove a directory DIR. The flag “-r” means recursively and “-f” means do so without asking for each file and subdirectory. Useful, but dangerous. Be careful!
• cd: Go to your home directory ($HOME)
• cd DIR: Change directory to DIR
• cd ..: Change directory to the parent directory of the current directory
• touch FILE: create an empty file with the name FILE

You also user “rm” to remove files;

rm file.txt

The command pwd tells you the current directory path.

[x_birbr@tetralith1 ~]$ mkdir mytestdir
[x_birbr@tetralith1 ~]$ cd mytestdir/
[x_birbr@tetralith1 mytestdir]$ mkdir testdir1
[x_birbr@tetralith1 mytestdir]$ mkdir testdir2
[x_birbr@tetralith1 mytestdir]$ mkdir testdir3
[x_birbr@tetralith1 mytestdir]$ rm -rf testdir3
[x_birbr@tetralith1 mytestdir]$ cd testdir1
[x_birbr@tetralith1 testdir1]$ touch file1.txt
[x_birbr@tetralith1 testdir1]$ touch file2.sh
[x_birbr@tetralith1 testdir1]$ touch file3.c
[x_birbr@tetralith1 testdir1]$ touch file4.dat
[x_birbr@tetralith1 testdir1]$ touch file5.txt
[x_birbr@tetralith1 testdir1]$ rm file5.txt 
[x_birbr@tetralith1 testdir1]$ 
[x_birbr@tetralith1 testdir1]$ cd ..
[x_birbr@tetralith1 mytestdir]$ cd testdir2/
[x_birbr@tetralith1 testdir2]$ 

b-an01 [~]$ cd mytestdir
b-an01 [~/mytestdir]$ cd testdir1
b-an01 [~/mytestdir/testdir1]$ 

cp - copy files/directories¶

This command is used to copy files or directories.

• cp myfile.txt DIR/: copy the file “myfile.txt” to the directory DIR
• cp DIR1/ DIR2/: copy the directory DIR1 into the directory DIR2 (Note: overwrites existing files with same name)
• cp -R DIR1/ DIR2/: copy the directory DIR1 and all subdirectories into the directory DIR2.

mv - rename files/directories¶

The command mv is used to rename files and directories. It can also be used to move a file or directory to another location.

• mv file1.txt file2.txt: renames file1.txt to file2.txt
• mv DIR1/ DIR2/: renames directory DIR1 to directory DIR2/
• mv file1.txt DIR1/: moves the file file1.txt into the directory DIR1/

Exercise¶

Symbolic links¶

Symbolic links are also called soft links, or just symlinks. It is a pointer to another file or directory.

• It is useful both for ease
  • you avoid using a long path each time you change to a directory, like your project directory
  • as well as to avoid changing hard links within other scripts or programs. This is good if you for instance install a program or use a script that assumes the library it uses is called libcoolness.a and not libcoolness.2.0.a. You can then just update the symlink instead of renaming the library or updating potentially many instances where it is mentioned in the program.

Command:

ln -s real-file-or-lib link-name

ln -s /proj/intro-linux/users/MYUSERNAME $HOME/myproj

[x_birbr@tetralith1 ~]$ ls -l
total 2
lrwxrwxrwx 1 x_birbr x_birbr   31 Sep 11 12:01 myproj -> /proj/intro-linux/users/x_birbr
drwxrwxr-x 4 x_birbr x_birbr 4096 Sep 11 11:43 mytestdir

Redirection¶

Usually, standard input comes from the keyboard etc. and the standard output goes to the screen. There is also standard error. All of these can be redirected with Linux commands.

• > redirects the output of some command
  • Example, output of “ls” to a file: ls > test.dat
• >> concatenate the output of some command to the content of a file
  • Example, adds the output of ls to the end of a file “test.dat”: ls >> test.dat
• < changes the standard input
• 2> redirects the standard error:
  • Example, redirect the error that is thrown from your program named “myprogram” to a file “error.log”: ./myprogram 2> error.log
• 2>&1 redirects both standard output and standard error
  • Example, redirect output and errors from your program to the same file: ./myprogram > logfile 2>&1

cat file >> file2

echo 'text to append_add_here' >> file2

printf "text to append\n" >> fileName

Pipes¶

Pipes are used when you want to take the output of one command and use it as input for another command.

Here follows some examples:

HINT: try the examples! You need a file file.txt for the examples. Either create a suitable one with some instances of the word string, or download (right-click and save): file.txt

grep -o -i string file.txt | wc -l

grep string file.txt > file.out

grep string file.txt >> file.out

Exporting variables¶

Environment variables store data that is used by the operating system and other programs.

Some are intrinsic to the operating system, some for a specific program/library/programming language, and some are created by the user.

The variables can both be used in scripts and on the command line. Usually you reference them by putting a special symbol in front of or around the variable name. By convention, environment variable names are in UPPER CASE.

Examples:

• $HOME Your home directory
• $PWD This variable points to your current directory
• $LD_LIBRARY_PATH a colon-separated list of directories that the dynamic linker should search for shared objects before searching in any other directories
• $OMP_NUM_THREADS Number of OpenMP threads
• $PYTHONPATH Path to the directory where your Python libraries and packages are installed

echo $ENVIRONMENT-VARIABLE

env

export VARIABLE=value

setenv VARIABLE value

Some examples:

export OMP_NUM_THREADS=8

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/your/custom/path/

echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/your/custom/path/" >> ~/.bashrc