Patterns and scripting¶

This section will look at finding patterns (grep, awk, wild cards, regular expressions) and scripting.

Here you will find descptions on how to search for files with specific patterns.

grep¶

This command searches for patterns in text files.

Syntax:

grep [options] [FILE]

FILE is the name of whatever file you want to look at.

Some commonly used options are:

• -i: ignore case
• -o: only output the instance of the pattern you search for, not the full line
• -e: use a regular expression as pattern for what you search for
• -c: do not output the line, just print a count of matching lines
• -n: Prefix each line of output with the 1-based line number within its input file.
• -r: Read all files under each directory, recursively. If no file operand is given, grep searches the working directory.

See man grep for more options.

Examples

grep 'word' FILE

grep -rine 'word' path/to/dir

grep -o -i string file.txt | wc -l

grep string file.txt > file.out

grep string file.txt >> file.out

find¶

The find command is used for file and directory search. You can search by name, size, modification time, or content.

Syntax

find [path] [options] [expression]

• path is the starting directory for the search
• options are settings or conditions for the search
• expression is the criteria for filtering and locating files (for instance, only search txt```files or only files nameddocs``).

Useful common options

• -type f: only search for files
• -type d: only search for directories
• -name NAME: only search for files with a specific name NAME or pattern
• -size [+/-]n: Searches for files based on size. +n finds larger files, -n finds smaller files. ‘n‘ measures size in characters.
• -mtime n: Finds files based on modification time. n represents the number of days ago.
• -exec command {} \;: Executes a command on each file found.

For more options, check man find

Examples

1. Find all files with extension .txt in the directory you are standing in and below:

find . -type f -name "*.txt"

find ../patterns/ -type f -name "*file*"

awk¶

Powerful, but somewhat more advanced command!

This command finds patterns in a file and can perform arithmetic/string operations. You can use it to transform data files and produce formatted reports.

It allows the user to use variables, numeric functions, string functions, and logical operators.

Things awk can do:

• Scan a file line by line
• Split each input line into fields
• Compare input line/fields to pattern
• Perform action(s) on matched lines

awk '/snow/ {print$1}' FILE

awk '{print $3 "\t" $4}' mydata.dat

awk '/r/ {print $2 "\t" $3}' mydata.dat

Wild cards¶

Wild cards are useful ‘stand-ins’ for one or more character or number, that you can use for instance when finding patterns or when removing/listing all files of a certain type.

Wild cards are also called “glob” or “globbing” patterns.

Common wildcards

• ? represents a single character
• * represents a string of characters (0 or more)
• [ ] represents a range
• { } the terms are separated by commas and each term must be a wildcard or exact name
• [!] matches any character that is NOT listed between the [ and ]. This is a logical NOT.
• ** specifies an “escape” character, when using a subsequent special character.

myfile?.txt

r*d

r[a,i,o]ck

a[d-j]a

[0-9]

cp {*.dat,*.c,*.pdf} ~

rm thisfile[!8]*

Regular Expressions¶

Regular Expressions are a type of globbing patterns that are used when you are working with text.

Regular Expressions can be used with programs like grep, find and many others.

Some common examples of regular expressions:

• . matches any single character. Same as ? in standard wildcard expressions.
• \ is used as an “escape” character for a subsequent special character.
• .* is used to match any string, equivalent to * in standard wildcards.
• * the proceeding item is matched zero or more times. ie. n* will match n, nn, nnnn, nnnnnnn but not na or any other character.
• ^ means “the beginning of the line”. So “^a” means find a line starting with an “a”.
• $ means “the end of the line”. So “a$” means find a line ending with an “a”.
• [ ] specifies a range. Same as for normal wildcards. This is an ‘or’ relationship (you only need one to match).
• | This wildcard makes a logical OR relationship between wildcards. You can thus search something or something else. You may need to add a ‘' before this command to avoid the shell thinking you want a pipe.
• [^] This is the equivalent of [!] in standard wildcards, i.e. it is a logical “not” and will match anything not listed within the square brackets.

$ cat myfile | grep '^s.*n$'

Scripting¶

Scripting is used to perform complex or repetitive tasks without user intervention. All Linux commands can be used in a script including wild cards.

The most common reason for making a script is probably to avoid writing the same command again and again.

#!/bin/bash
grep 'ABCD' file.dat > file_filtered.dat
./program.sh < file_filtered.dat > output.dat

$ chmod u+x analysis.sh

$ ./analysis.sh

#!/bin/bash
find $HOME/exercises/script/image -type f -name "*.png" -exec cp {} myimages \; 
find myimages -type f -name "*er*" > someimagesfiles.txt

More advanced example¶

#!/bin/bash
find . -type f -name "*.eps" -exec cp {} $HOME/figures \;
FIGFILES="$HOME/figures/*"
for f in $FIGFILES
do
    g=${f%.*}
    ps2pdf -DEPSCrop "$f" "$g.pdf"
done 

For more examples of (more useful) scripts, see for instance this list of 25 Easy Bash Script Examples.