Advanced Unix Programming

Bibliography

Chappter 1

Introduction to UNIX System

Simple View of an Operating System

In the developments of computers most commendable developments can be mentioned as the development of operating systems with the help of which a lay man is also in a position to avail the services of computers without joining computer science majors! An Operating System is the SW layer between the hardware and user as shown in the following figure and gives clean and easy interface to the user.

Figure 1.1

An Operating System is responsible for the following functions :

►   device management using device drivers,

►   process management using processes and threads,

►   inter-process communication,

►   memory management,

►   file systems.

In addition, all operating systems come with a set of standard utilities. The utilities allow common tasks to be performed such as :

►   being able to start and stop processes,

►   being able to organize the set of available applications,

►   organize files into sets such as directories,

►   view files and sets of files,

►   edit files,

►   rename, copy, delete files and

►   communicate between processes.

Kernel

The kernel of an operating system is the part responsible for all other operations. When a computer boots up, it goes through some initialization functions, such as checking memory. It then loads the kernel and switches control to it. The kernel then starts up all the processes needed to communicate with the user and the rest of the environment (e.g. the LAN).

The kernel is always loaded into memory, and kernel functions always run, handling processes, memory, files and devices.

The traditional structure of a kernel is a layered system, such as Unix. In this, all layers are part of the kernel, and each layer can talk to only a few other layers. Application programs and utilities live above the kernel.

The Unix kernel looks like ( Fig. 1.2).

Figure 1.2 UNIX Kernel

Most of the Operating Systems being built now use instead a micro kernel, which minimizes the size of the kernel. Many traditional services are made into user level services. Communication being services is often by an explicit message passing mechanism.

The major micro-kernel Operating System is Mach. Many others use the concepts of Mach (see Fig. 1.3).

Figure 1.3 Micro Kernel Architecture

Some systems, such as Windows NT use a mixed approach ( Fig. 1.4).

Figure 1.4

Distinguished Applications

An Operating System has been described as an application with no top (B. Meyer, Object-oriented Software Construction). Other applications interact with it, through a large variety of entry points. In order to use an OS, you need to be supplied with at least some applications that already use these entry points.

All Operating Systems come bundled with a set of utilities which do this. For example :

►   Windows95 has a shell that allows programs to be started from the Start button. There is a standard set of applications supplied.

►   MSDOS starts up with COMMAND . COM to supply a command line prompt, and a set of utilities.

►   Unix has a set of command line shells and a huge variety of command line

utilities.

►   X-Windows supplies a login shell (xdm). Others supply file managers, session managers, etc which can be used to provide a variety of interfaces to the underlying Unix/POSIX system.

Command Interpreter

When a user interacts with an Operating System they always do so through the intermediary of a command interpreter. This responds to user input in the following ways :

►   it starts applications,

►   it stops applications,

►   it allows the user to switch control between applications and

►   it may allow control over communication between an application and other applications or the user.

The command interpreter may be character based, as in the MSDOS COMMAND . COM or the Unix shells. It may be a GUI shell, such as the Windows 3.1 Program Manager.

The interpreter may be simple, or can have the power of a full programming language. It may be imperative (as in the Unix shells), use message passing (as in AppleScript) or use visual programming metaphors such as drag-and-drop for object embedding (as in Microsoft’s OLE).

It is important to distinguish between the command interpreter and the underlying Operating System. The command interpreter may only use a subset of the capabilities offered by the Operating System; it may offer them in a clumsy or sophisticated way; it may require complex skills or be intended for novice.

Introduction to Unix File System

Files are stored on devices such as hard and floppy disks. OS defines a file system on the devices. Many OS use a hierarchical file system.

A directory is a file that keeps a list of other files. This list is the set of children of that directory node in the file system. A directory cannot hold any other kind of data.

On MSDOS a file system resides on each floppy or partition of the hard disk. The device name forms part of the file name.

On Unix there is a single file system. Devices are mounted into this file system. (Use the command mount to see this.)

Figure 1.5 Hierarchical File Systems

File and Directory Naming

An individual node of the file system has its own name. Naming conventions differ between operating systems. In MSDOS, a name is constructed of upto 8+3 characters. Windows95 uses tricks on top of the MSDOS file system to give long file names of upto 255 characters. In standard Unix (POSIX) a name may consist of upto 256 characters.

The full file names are constructed by concatenating the directory names from the root down to the file, with some special separator between names. This is known as absolute path naming. In MSDOS, the full path name also includes the drive name.

Example : MSDOS

C : \expsys\lectures\search.txt

Example : Unix

/usr/usrs/os

/usr/usrs/os/myfile

Relatiue naming means that files are named from some special directory :

.   current directory (Unix and MSDOS)

..   parent directory (Unix and MSDOS)

~   home directory (some Unix shells)

~   user home directory of user (some shells)

Example : Unix

~fred/../bill/dirl/./../filel

If just the name itself is given without any special prefixes (such as /, ., .., ~) then it refers to the file in the current working directory.

Command history

Bash has a command history for convenience. The list of previous commands may be obtained by executing the following command.

history

!n

(n is an integer) will re-execute the nth command.

! !

This executes the most recent command

! cp

This executes the most recent command which starts with cp.

Up arrow, down arrows can be used in some shells to recollect the commands from command history buffer.

man Pages

All the Unix command information are organized in a special fashion like the following :

►   The user-level commands are all in Section One.

►   Section Two is the Unix Application Programmer’s Interface, API (i.e., C functions directly supported by Unix).

►   Section Three is library extensions to these.

►   Section Four defines devices known to Unix.

►   Section Five defines common file formats.

►   Sections Local and New are for stuff we have added to our local system.

If we run man command with a name first it will check for commands with that name.

Example

man sleep

man 2 sleep

This displays details of sleep library function if available

man 3 sleep

This displays details of sleep system call if exists any.

apropos command can be used to displays names of all the commands whose manual page contains a search pattern.

Example

apropos TERM

This displays names of the Unix commands, system calls or library functions whose manual page contains the search pattern TERM.

cat Command

This is used to create files.

Example

cat > ABC

This is a test file.

I wish you find happy to create first file.

^ d

This is also used to see the file(s) content. If the file contains more lines then it simply scrolls the matter of that file.

Example

cat ABC

This command is also used to create duplicate of a file.

Example

cat ABC >XYZ

or

cat XYZ

XYZ becomes duplicate copy of file ABC.

This cat command can be used to see the content of more than one file.

Example

cat ABC XYZ

This cat command can be used to join the content of two or more files and create another file.

Example

cat ABC XYZ > MNO

Now MNO file contains the contents of both file XYZ and ABC.

While joining two or more files and creating a combined file we can add interactive input also.

Example

cat ABC - XYZ > PPP

You type what ever you want followed by CTRL d at the end.

^ d

Now file PPP contains content of ABC, the interactive input and the content of file XYZ in the same order. By changing the location of we can add interactive input between any two files.

Chappter 2

The vi Editor

This is the popular editor in Unix since last 30 years. This is a screen editor which is based on another editor known as elvis. It has three modes: Input Mode in which what ever user typed will be written into the document. Command Mode is the one in which the user can enter commands. To move from Input mode to command mode, we have to press ESC key. If you are already in command mode and when we press ESC key we will get beep sound. This command mode is also called as ESC mode. The third mode is called as colon mode in which also users can run commands to do some editing on the document content. Ofcourse, there are some people who debate that this is not a separate mode!! In a nutshell, the following commands are summarized to immediately work under Unix.

1.   vi filename Opens vi editor with the given filename.

2.   When the editor opens a screen will be opened with the command mode.

3.   To enter text press i then input mode will be displayed at bottom right part of the screen.

4.   To stop typing, press Esc key. Then command mode comes. Press

: w   to save the matter and resume editing.

: wq   to save the matter and quit the vi editor.

: q!   to quit the editor without saving.

5.   Three modes are present in vi editor namingly

(i)   Command mode,

(ii)   Input mode,

(iii)   Colon mode.

6.   In command mode commands can be entered. By pressing Esc key one can go to command mode from other modes.

(a)   press i to insert text before the current cursor position.

(b)   press I to insert text at the beginning of the line.

(c)   press a to insert text after the cursor position.

(d)   press A to insert text at the end of the current line.

(e)   press o to open a new line below the current line.

(f)   press O to open a new line above the current line.

(g)   press r to replace the present character with a character.

(h)   press R to replace a group of characters from current cursor position.

(i)   press x to delete present character.

(j)   press J to join the next line to the end of the current line.

(k)   press dd to delete the current line.

(l)   press 4dd to delete 4 lines from the current line.

(m)   press dw to delete the current word.

(n)   press 7dw to delete 7 words from the current word onwards.

(o)   press 30i*Esc (invisible command) to insert 30 *’s at the cursor position.

(p)   press u to undo the effect of the previous command on the document.

(q)   press . to repeat the previous command.

(r)   press yy to copy the entire line in to the buffer.

(s)   press yw to copy the entire word in to buffer.

(t)   press p to place the copied or deleted information below the cursor.

(u)   press P to place the copied or deleted information above the cursor.

7.   Colon mode commands

(a)   Search and substitute commands

1.   :/raja searches for the string raja in the forward direction, press n to repeat the search.

2.   :?raja search for the string in the backward direction press n to repeat the above search.

3.   :s/raja/rama replaces the first occurrence of raja with rama.

4.   :s/raja/rama/g replaces all raja’s with rama in the present line.

5.   :g/raja/s/raja/rama/g replaces all raja’s by rama in whole file.

(b)   Block delete commands

1.   :ld delete the line 1.

2.   :l,5d deletes the lines from 1 to 5.

      $ Means last line of the file.

      .Means present line (i.e) present line.

3.   :10,$d deletes ines from 10th line to the last line of the file.

4.   :l,$d deletes ines from 1 to last line of the file.

5.   :.,$d deletes ines from present line to the last line.

6.   :.-3,.d deletes the lines from present line and above 2 lines (deletes 3 lines including the cursor line).

7.   :.,. + 4d deletes the lines from the present cursor line followed 3 lines (total 4 lines).

8.   :-l,. + 3d deletes the lines one above the cursor line followed by it 3 lines.

9.   :18 cursor goes to 18th line of the file.

(c)   Block copy commands

1.   : 1,5 co 10 copies the lines from 1 to 5 after 10th line.

2.   :1,$ co $ copies the lines from 1 to last line after last line.

3.   :.,.+5 co 8 copies lines from present line to 5 lines after 8th line.

4.   :-3,. co 10 copies the lines from present cursor line and above 3 lines after 10th line.

(d)   Block moving commands

1.   :1,5 mo 9 moves lines from 1 to 5 after 9th line.

2.   :1,$ mo $ moves lines from 1 to $ after last line.

3.   :.,. + 5 mo 10 moves lines from present line and next 5 lines after 10th line onwards.

4.   :.-3,.mo 10 moves present line and above 3 lines after 10th line.

Importing & Exporting the files

1.   :1,5 w filename writes lines 1 to 5 in the specified filename.

2.   :1,5 w! filename overwrites lines 1 to 5 in the specified filename.

3.   :r filename adds the content of filename after the current line.

8.   Book mark command

Bookmarks (markers) are not visible and are useful to jump from one line to another quickly. Markers should be in lower case. To have the marker on a specified line press m followed by a lower case alphabet (say a) then marker for that line is set as a. To go to the marked line press ‘a (‘back quote) followed a. eg: go to 500 th line, press mb (b is the marker). To go to the 500th line from anywhere in the document press b. Then the cursor goes to the 500th line.

Chappter 3

Redirection Operators

Standard Input, Output Redirection Operators

Unix supports input, output redirection. We can send output of any command to file by using > operator.

Example

command>aaa

Output of the given command is sent to the file. First, file aaa is created if not existing, otherwise its content is erased and then output of the command is written.

cat aa>aaaa

Here, aaaa file contains the content of the file aa.

We can let a command to take necessary input from a file with < operator (standard input operator).

cat

This displays output of file aa on the screen.

cat aa aa1 aa3>aa12

This creates the file aal2 which contains the content of all the files aa, aal and aa3 in order.

cat as

This makes cat command to take input from the file aa and write its output to the file as. That is, it work like a cp command.

Unix has a nice (intelligent) command line interface. Thus, all the following commands work in the same manner.

cat as

cat >as

as

as cat

>as cat

>as

This discussion is meaningful with any command. For example, consider the following C program which takes these integers and writes their values.

#include

void main()

{

int x,y,z;

scanf(%d%d%d, &x, &y, &z) ;

printf(%d\n%d\n%d\n, x, y, z);

}

Let the file name be a. c and by using the either of the following commands, its machine language file a is created.

gcc-o a a.c

cc-o a a. c

When we start this program a by simply typing a at the dollar prompt, it takes 3 values and displays given values on the screen.

a>res

This program takes these values interactively and writes the same into file res. You can check by typing cat res.

a

cat as

cat >as

as

as cat

>as cat

>as

This command takes necessary input from the file res and displays the results on the screen.

a as

a >as

as

as a

>as a

>as

Each of these commands takes three values from the file res and write the same in the file as.

Similarly, >> operator can be used to append standard output of a command to a file.

Example

command>>aaa

This makes, output of the given command to be appended to the file aaa. If the file aaa is not existing, it will be created a fresh and then standard output is written.

Here the document operator (<<)

This is used with shell programs. This signifies that the data here is rather in a separate file.

Example

grep Rao<

I like PP Reddy

I know Mr. PN Rao since 1987

I wanted to see Raj today

Mr. Rao, please see me today

end

The above sequence of commands when executed at the dollar prompt, we will get those lines having Rao as output of grep command. Here, by using << operator we are mentioning that the data is directly available here.

more Command

This command is used to see the content of the files page by page or screen by screen fashion. This is very useful if the file contains more number of lines.

Example

more filenames(s)

more file1 file2

This displays content of the files filel and f ile2 one after another.

more

This also displays the content of the filel in screen by screen fashion.

more file1 file2 ... filen > XXX

This command creates file XXX such that it contains the content of all the given files in the strictly same order.

more + /rao filename

This command displays the content of the given file starting from the line which contains the string rao.

more +10 filename

This command displays the content of the file from 10th line.

pg command

This command is also used to see the content of the files in page by page fashion. However, this is not available in recent versions. Rather more command is in wide use and is more flexible.

nl command

This command is used to display the content of the file along with line numbers.

Example

nl filename

tail command

tail filename(s)

This command displays last 10 lines of the given file(s).

tail -1 filename(s)

This command displays last 1 line of the given file(s).

tail +2 filename(s)

This command displays second line to last line of the given file(s)

head command

head filename(s)

This command displays first 10 lines of the given file(s).

head -2 filename(s)

This command displays first 2 lines of the given file(s)

mkdir command

This is used to create a new directory.

mkdir rao

This creates rao directory in the current directory.

mkdir /tmp/rao

This creates rao directory in /tmp directory.

mkdir /bin/rao

This fails for normal users because of permissions (/bin belongs to super user).

rmdir command

This is used to remove empty directory only.

rmdir rao

This removes rao directory of current working directory.

rmdir /tmp/rao

This removes rao diretcory in /tmp directory.

pwd command displays where currently we are located.

cd directoryname

This changes the current working directory to the given directory.

cd

This command takes you to your home directory.

Is command

This command displays names of the files and directories of current directory.

a1

a2

a3

a4

a5

The following command displays names of files and directories of current directory in long fashion. That is, file permissions, owner name, group, links, time stamps, size and names.

In Unix, files whose names start with . are called as hidden files. If we want to see their details also, then we have to use -a option (ofcourse either alone or with other options).

For example, the following command displays other files also whose names starts with'.'.

The following command displays details of the files in chronological order.

Is -l filename

It displays only that file details if it exists.

Is -l directoryname

It displays the files and directory detials in the given directory.

All the options -a, -t etc can be also used. Moreover, Unix commands will be having excellant command line interface. Thus, all the following commands are equivalent.

Is -a -1 -t

Is -alt

Is -a -t -1

Is -atl

Is -1 -a -t

Is -lat

Is -1 -t -a

Is -lta

Is -t -1 -a

Is -tla

Is -t -a -1

Is -tai

-R option with Is command displays details of files and subdiretcories recursively.

Example

Is -alR I (ofcourse, you can go for a cup of coffee and come back before you see the prompt again!!)

This command displays all the files in Unix system.

cp command

cp command is used to duplicate a file(s).

Syntax

cp source destination

mv command

mv command is used to move file(s) from one directory to another directory or to rename the file.

The options include

-i interactive confirmation of overwrites

-f force a copy

-R recursively copy to a directory

Syntax

mv source destination

The options include

-i interactive confirmation of overwrites

-f force a move

wc command

wc filename or wc< f i 1 ename

This command displays number of lines, words and characters in the given file.

wc -1 filename

This displays number lines in the given file.

wc -w filename

This displays number words in the given file.

wc -c filename

This displays number characters in the given file.

find command

This command is used to locate files in the Unix directory tree.

Find directoryname-name filename to be found

Example

find / -name core

This command displays all the occurrences of the file named core under / directory.

find, -ctime 2 -name

This command displays names of those files which are created in the last two days and are in the current directory.

find . -mtime 2 -name

This command displays names of those files which are modified in the last two days and are in the current directory.

find . -size 10 -name

This command displays names of those files whose size is greater than 10 blocks of size 512 bytes and are in the current directory.

find . -type d -name

This command displays names of directories in the current directory.

Link Files

Unix supports two types of links (shortcuts) for files and directories namingly hard links and symbolic links.

Example

In al a6

Here, a6 becomes hard link to the file al. Whatever operations we do on a6 is really seen from al also. The reverse also is true. In fact, a6 will not take extra disk space. If we delete al (or a6) yet the file content is accessible through other name.

Hard links cannot be created to directories. Moreover, they cannot be created to the files of other partitions.

Is -1 al a6 gave the following result

In al a7

Is -al a1 a6 a 7 gave the following result

We can observe that link count is increasing whenever a new hardlink is created for a file. Similarly, whenever we remove a hardlink file link count is reduced.

rm a6

Is -1 al a 7 gives results

I-node numbers or hardlink and original files are same.

Is -li al a7

Symbolic Links

In -s al a8

Is -1 al a8

We can see the difference. Though, whatever operations we do on symbolic link really takes place on the original file yet if we delete original file the information of the file cannot be accessible through symbolic link unlike hardlink files. Ofcourse, if we delete symbolic link yet the information is accessible through original name. Moreover, i-node numbers or orginal file and symbolic link files are different. In fact, symbolic link file will take seperate disk block in which path of the original file is saved.

Is -li al a8

Main advantage of symbolic link files is that they can be used to create links for directories and also to the files of other partitions. In fact, symbolic links are used for SW fine tuning. For example check for file ‘x’ in Linux system, which is normally symbolic link to the appropriate X server (Check in /usr/XHR6/bin).

Is -1 /usr/XHR6/bin/X gave me the following results

lrwxrwxrwx 1 root root 7 Feb 7 06:31 /usr/XHR6/bin/X -> XFree86

If we want to change to some other X server, simply we change X to point to that and start the X server.

Chappter 4

Filters

uniq command

This command displays uniq lines of the given files. That is if successive lines of a file are same then they will be removed. By default, the output will be on to the screen. This can be used to remove sucessive empty lines from the given file.

cat list-1 list-2 list-3 | sort | uniq final.list

Concatenates the list files, sorts them, removes duplicate lines, and finally writes the result to an output file.

The useful -c option prefixes each line of the input file with its number of occurrences.

Let the file testfile contains the following lines.

This line occurs only once.

This line occurs twice.

This line occurs twice.

This line occurs three times.

This line occurs three times.

This line occurs three times.

Then, the following command is executed the result is as displayed below.

uniq —c testfile

1 This line occurs only once.

2 This line occurs twice.

3 This line occurs three times.

Similarly, when the following command is executed the result is displayed as below.

sort testfile | uniq -c | sort -nr

3 This line occurs three times.

2 This line occurs twice.

1 This line occurs only once.

grep command

This command is used to select lines from a file having some specified string.

grep rao xyz

This displays those lines of the file xyz having string rao.

grep [rR]ao xyz

This displays those lines of the file xyz having strings either Rao, or rao.

grep [rR]a[uo] xyz

This displays those lines of the file xyz having strings either Rao, or Rau, or rao , or rau .

grep ^rao xyz

This displays those lines of the file xyz which starts with string rao

grep rao$ xyz

This displays those lines of the file xyz which ends with string rao.

grep ^rao$ xyz

This displays those lines of the file xyz which contains the string rao only. No more characters in the line.

grep ^$ xyz

This displays empty lines of the file xyz.

grep ^[rR]ao xyz

This displays those lines of the file xyz which starts with either Rao or rao.

grep [rR]ao$ xyz

This displays those lines of the file xyz which ends with Rao or rao.

-n option if we use with grep command it displays line numbers also,

grep -n rao xyz

This displays those lines of file xyz which are having the string rao along with their line numbers.

-v option if we use with grep command it displays those lines which does not have the given search pattern.

grep -v rao xyz

This displays those lines of the file xyz which does not contain the string rao.

fgrep (fixed grep) and egrep (extended grep) commands

fgrep is used to search for a group of strings. One string has to be seperated from the other by a newline.

$fgrep 'rao

>ram

>raju' filename

This command displays those lines having either rao or ram or raju.

fgrep will not accept regular expressions.

egrep is little more different. It also takes a group of strings. While specifying strings piping (|) can be used as seperator.

Example

egrep 'rao|ram|raju' filename

In addition, it accepts regular expressions also.

cut command

cut -f1,3 filename

This displays 1st and 3rd words of each line of the given file. Between word to word TAB should be available.

cut -d : -f1,3 /etc/passwd

This displays username, UID of each legal user of the machine. Here, with -d option we are specifying that : is the field seperator between word to word.

Cut command cannot change the natural order of the fields. That is, the following command also gives same result as that of the above command.

cut -d : -f3,l /etc/passwd

cut -d: -fl-3 filename

This displays 1st word to third word from each line of the given file.

cut -f: -f3- filename

This displays 3rd word to till last word of each line of the given file.

cut -c3-5 filename

This displays 3rd character to 5th character of each line of the given file.

cut -d: -fl /etc/passwd > al

File al contains usernames of legal users of the machine.

cut -d: -f3 /etc/passwd > a3

File a3 contains UID’s of each legal user of the machine.

paste command

This is used to join files