Shell Scripting

Basics

Going to previous directory(in hierarchy) is cd .. but going back to

the folder location before the current one (back button functionality
is cd ~ or cd
Running background jobs.
Just put & after the command eg: uncompress abc.tar &
If the background jobs need to take an input and if it gives an
output, then use files (for input and output)
Nice command (priority: -20 to 19: lowhigh)
o Nice --10 <command name> (sets priority to -10)
o renice -4 -p 3423 (this will set the priority of process id no
3423 to -4, which will inturn increase its priority over others)
o renice 14 -u sarath,satish -g custom (this will set all
process owned by "sarath","satish" and also the group
"custom" to 14)
o nice (returns the priority of the current shell running.
Escaping
o Use \ or quotes to escape
o Eg: echo 2 * 3 > 5 is so true lol
o Eg: echo 2 \* 3 \> 5 is so true lol
Command is more than a line:

Know your control commands (stty a)

If by chance your screen output gets stuck at any point (u have

pressed CTRL+S by mistake) resume it by pressing CTRL+Q

Wildcards
Recall what below stuff mean
1. * , ? , [ ] , [! ]
2. If the string doesnt match anything it will leave the string with the value
untouched
3. [a-zA-Z0-9-_]
4. [!a-zA-Z] matches any character that isnt a letter. To match ! itself, place
it after the first character in the set or precede it with backslash \ eg: [\!]
5. Brace Expansion {}
B{ar{d,n,k},ed}s
6. Difference between using [] and {}

Unix I/O
1. Takes sequence of characters (bytes)
2. Everything that accepts or produces data is treated as a file
3. Unix has a single way of accepting
a. Output as Standard Output
b. Input as Standard Input
c. Error as Standard Error or Standard Error Output
4. Commands usually take arguments. But lets say they dont take any
arguments then,
a. For input : command < filename
b. For output : command > filename
c. Both sides: cat < filename1 > fileName2
5. Pipe : cut -d: -f1 < /etc/passwd | sort | lp

Shell utilities
CAT
Syntax:
cat filename

cat options filename

cat file1 file2

cat file1 file2 > newcombinedfile

Display text files on screen
$ cat test01.txt
$ cat /etc/passwd > /tmp/test04.txt

Copy text files

$ cat file03.txt > newfile03.txt
-sh-4.1$ cat test01.txt - test02.txt
This is

a test01 test file

This is the second line in test01

---nice!! I can type here :)

---nice!! I can type here :)
--now i have to presss CTRL+D--now i have to presss CTRL+D

This is a test02 test file

This is the second line in test02

Combine text files

$ cat test01.txt test02.txt test04.txt
$ cat test01.txt test02.txt test04.txt > output.txt
$ cat /etc/passwd | less

Create new text files

-sh-4.1$ cat > abc.txt
I just created a file using cat command.
Oh yes!! I have to press ctrl+d to save the file
-sh-4.1$ vi abc.txt
-sh-4.1$ cat >> abc.txt
Now I am trying to append the file using >> operator.
Lets see how it works.

-sh-4.1$ cat abc.txt

I just created a file using cat command.

Oh yes!! I have to press ctrl+d to save the file
Now I am trying to append the file using >> operator.
Lets see how it works.
With options

-A, --show-all
equivalent to -vET
-b, --number-nonblank
number nonblank output lines
-e

equivalent to -vE

-E, --show-ends
display $ at end of each line
-n, --number
number all output lines
-s, --squeeze-blank
never more than one single blank line
-t

equivalent to -vT

-T, --show-tabs
display TAB characters as ^I
-u

(ignored)

-v, --show-nonprinting
use ^ and M- notation, except for LFD and TAB

GREP
1. grep search-string filenames

2. With regular expression

SORT
1 Basic command usage
$ sort
b
z
a
w
s

And here is the output :

a
b
s
w
z

So we see that the output produced was in sorted form.

2. Sort numbers
In the following example, I filled a text file (sort.txt) with some random numbers.
$ cat sort.txt
8
2
6
1
5
3

Then I used the sort command with sort.txt as input file to the command.
$ sort sort.txt
1
2
3
5
6
8

So we see that sorted list of numbers was produced in output.

3. Sorting words
In this example, the sort.txt file is filled with some words.
$ cat sort.txt
UK
Australia
Newzealand
Brazil
America

Now, this file is given as input to the sort command:

$ sort sort.txt
America
Australia
Brazil
Newzealand
UK

So we see that words were sorted according to dictionary ordering. Even the
words beginning with same alphabet were sorted according to succeeding
alphabets.

4. Use sort to directly write data in sorted manner

This command can be used to write unsorted input data to a file directly in
sorted manner.
Here is how this can be done :
$ sort > sort.txt
9
Hello
4
Why
8

Bye

After the above operation, let's check the file contents :

$ cat sort.txt
4
8
9
Bye
Hello
Why

So the output suggests that the input was first sorted and then written to file.

5. Write sorted concatenation of all input files to standard output

If more that one file is provided as input, the sort command produces a sorted
concatenation on stdout.
Here is an example:
$ cat sort1.txt
7
4
9
1
$ cat sort2.txt
8
5
6
2

Here is the output :

$ sort sort1.txt sort2.txt
1
2
4
5
6
7
8
9

So we see that a sorted concatenation was produced in output.

6. Write result of sort in a file

The output of sort command can be written to a file by using -o option.
Here is how it's done :
$ sort -o sort.txt
4
9
2
8
1

Now let's check the file :

$ cat sort.txt
1

2
4
8
9

So we see that the output was actually written to the file whose name was
supplied as input to sort through -o option.

7. Sort months
There is an interesting option -M through which the month names can be sorted.
Here is an example :
$ sort -M > sort.txt
DEC
JAN
FEB

Now, let's check the file contents for output :

$ cat sort.txt
JAN
FEB
DEC

So we see that sort command actually sorted the month names.

8. Sort human readable numbers

Another interesting option -h is provided by sort command through which human
readable numbers.
Here is an example :
$ sort -h > sort.txt
2G
1K
3M

Now, let's check the file for output:

$ cat sort.txt
1K
3M
2G

So we see that the numbers were sorted.

9. Produce reverse sorted results

Using -r option provided by sort command, the results can be produced in
reverse order.
$ sort -h -r > sort.txt
2G
1K
3M

Here is the output of file :

$ cat sort.txt

2G
3M
1K

So we see that this time the sorting results were written in reverse sorted order.

10. Compare according to string numerical value

This can be done using -N option.
Here is the input :
$
7
4
9
1

cat > sort.txt

mangoes
oranges
grapes
apple

$
1
4
7
9

sort -n sort.txt
apple
oranges
mangoes
grapes

Here is the output :

CUT
Reference: http://www.computerhope.com/unix/ucut.htm
Default delimiter is tab
-f
cut -f 3 data.txt
cut -f 2-4 data.txt
cut -f 1-2,4-5 data.txt
cut -f 3- data.txt
-c
cut -c 3-12 data.txt
-b
cut -b 3-12 data.txt
Specifiying delimiter
cut -f 1 -d ':' /etc/passwd
cut -f 1,3 -d ':' /etc/passwd
cut -f 1,3 -d ':' --output-delimiter=' ' /etc/passwd
cut -f 1,3 -d ':' --output-delimiter=$'\t' /etc/passwd

SEED
1. Deleting lines in a file using sed command
1.1. Deleting the first line in the input file
bash-3.00# sed '1d' textfile.txt
This is line 2
This is line 3
This is line 4
This is line 5
1.2 Deleting the last line in the input file
bash-3.00# sed '$d' textfile.txt
This is line 1
This is line 2
This is line 3
This is line 4
bash-3.00#
1.3 Deleting arbitary lines in the input file
To delete the lines 2 through 4 in the above file, use the following sed
command
bash-3.00# sed '2,4d' textfile.txt
This is line 1
This is line 5
the 'd' flag is used to delete lines, though the single quote is not
mandatory in the above commands (you can simply use sed 1d textfile.txt
to delete the first line in the input file), its advisable to use it as a good
practice and will be useful when you use the -e option at times to execute
a series of sed commands inline.

2. Replace or substitute all occurrences of a pattern using sed

command
In the above text file, if you need to replace all occurrences of the
pattern "line" with the text "line number", the following sed command
can be used (the 's' flag in the sed command is used to substitute text
which matches a pattern).
bash-3.00# sed 's/line/line number/g' textfile.txt
This is line number 1
This is line number 2
This is line number 3

This is line number 4

This is line number 5 Note: The /g flag in the above command is used to
instruct sed to substitute globally i.e all occurrences of the input pattern in
the line will be replaced and as a result all words matching the pattern in
the input file will be substituted by the other, without /g only the first word
matching the pattern in a line will be replaced.
3. Delete all occurrences of a pattern in a file using sed command
This is a simple application of the /s flag where the target string is
none.
For example, to remove all occurrences of word "line" in the above file,
the sed command would be
bash-3.00# sed 's/line//' textfile.txt
This is 1
This is 2
This is 3
This is 4
This is 5
Note that the word to be replaced upon matching the pattern is none in
the above command ('/s/line//')
To remove all occurrences of the character space in the above input file,
the sed command would be
bash-3.00# sed 's/ //g' textfile.txt
Thisisline1
Thisisline2
Thisisline3
Thisisline4
Thisisline5
Also note that if you don't use single quotes in the above sed command,
then the space pattern should be enclosed in quotes for the command to
work.
bash-3.00# sed s/' '//g textfile.txt
Thisisline1
Thisisline2
Thisisline3
Thisisline4
Thisisline5

4. Deleting lines which matches a pattern using sed command

To delete lines which matches a pattern, use the /d flag along with the
pattern to be matched, for example, to delete lines matching the
pattern "line 1" in the above file, the command would be
bash-3.00# sed '/line 1/d' textfile.txt
This is line 2
This is line 3
This is line 4
This is line 5
Now using the above commands, one can also delete the first and last line
in a file which matches a pattern.
5. Deleting the first line which matches a pattern using sed command
bash-3.00# sed '/line/{1d;}' textfile.txt
This is line 2
This is line 3
This is line 4
This is line 5
In the above command, '1d' is used in braces to indicate that its a
command (to delete the specific line, which matches a pattern), this is the
way to separate sed flags in command/scripts.
6. Deleting the last line which matches a pattern using sed command
bash-3.00# sed '/line/{$d;}' textfile.txt
This is line 1
This is line 2
This is line 3
This is line 4
TR
1. Convert lower case to upper case

The following tr command is used to convert the lower case to upper case
$ tr abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ
thegeekstuff
THEGEEKSTUFF

The following command will also convert lower case to upper case
$ tr [:lower:] [:upper:]
thegeekstuff
THEGEEKSTUFF

You can also use ranges in tr. The following command uses ranges to convert lower to upper
case.

$ tr a-z A-Z
thegeekstuff
THEGEEKSTUFF

2. Translate braces into parenthesis

You can also translate from and to a file. In this example we will translate braces in a file with
parenthesis.
$ tr '{}' '()' < inputfile > outputfile

The above command will read each character from inputfile, translate if it is a brace, and
write the output in outputfile.
3. Translate white-space to tabs

The following command will translate all the white-space to tabs

$ echo "This is for testing" | tr [:space:] '\t'
This
is
for
testing

4. Squeeze repetition of characters using -s

In Example 3, we see how to translate space with tabs. But if there are two are more spaces
present continuously, then the previous command will translate each spaces to a tab as
follows.
$ echo "This
This

is

for testing" | tr [:space:] '\t'

is
for
testing

We can use -s option to squeeze the repetition of characters.

$ echo "This
This
is

is
for

for testing" | tr -s [:space:] '\t'

testing

Similarly you can convert multiple continuous spaces with a single space
$ echo "This is for testing" | tr -s [:space:] ' '
This is for testing

5. Delete specified characters using -d option

tr can also be used to remove particular characters using -d option.

$ echo "the geek stuff" | tr -d 't'
he geek suff

To remove all the digits from the string, use

$ echo "my username is 432234" | tr -d [:digit:]
my username is

Also, if you like to delete lines from file, you can use sed d command.
6. Complement the sets using -c option

You can complement the SET1 using -c option. For example, to remove all characters except
digits, you can use the following.
$ echo "my username is 432234" | tr -cd [:digit:]
432234

7. Remove all non-printable character from a file

The following command can be used to remove all non-printable characters from a file.
$ tr -cd [:print:] < file.txt

8. Join all the lines in a file into a single line

The below command will translate all newlines into spaces and make the result as a single
line.
$ tr -s '\n' ' ' < file.txt

Shortcuts
Cursor movement
By letter
a.
b.
c.
d.

Ctrl + B : one character back without deleting

Ctrl + F : one character front without deleting
Ctrl + D : delete character where the cursor is pointing
DEL
: same as above

a.
b.
c.
d.

ESC + B : one character back without deleting

ESC + F : one character front without deleting
ESC + D : delete character where the cursor is pointing
CTRL + Y : retrieve the last item killed. (its like UNDO)

By Word

By Line
a. CTRL + A : moves to the beginning of the line
b. CTRL + E : moves to the end of the line
c. CTRL + K : kill rest of the line from cursor point (including cursor
point)