Archive Utilities

Learning Objectives:
1.

2.

To introduce different archive utilities for file compression with their usage To compare & summarized different archive utilities

COMP111 Lecture a1 / Slide 2

Archive Utilities
Table of Content
Compress gzip compression Zip compression (PC Compatible) Zip (PC pkzip Compatible) Unzip (PC pkunzip Compatible) tar archive

No compression Selective restore With Compression

Cpio ( selected reference ) Summary

COMP111 Lecture a1 / Slide 3

Need for Archive Utilities

Archive refers to backup (with or without data compression) Data compression saves disk space / transmission time Combine multiple files/directory into one file - easier to maintain / transfer (download) Extra integrity checking (CRC) Regular backup helps

Recovery from disasters (disk crash) Human errors (accidentally deleting files / directory) Software errors that corrupts data / files (including virus)

Good idea to keep multiple generations and strategic snapshots (e.g. year end / month end) of backups Backup devices tapes, magnetic-optical (MO) disks, CDR/CDRW, (disk files usually not very good)

COMP111 Lecture a1 / Slide 4

Summary
Archive Restore List Usage Notes

compress
(*) gzip

uncompress
gzip -d gzip -l

Compress single file

Backward compatible with compress Compress (better ratio) single file

(*) zip (-r)

tar cvf

unzip
tar xvf

unzip -v
tar cvf

DOS pkzip / Winzip compatible Compress/backup multiple files/ dir.

Backup multiple files/ dir. (cf. gtar (*))

cpio -ocvB cpio -icvdB cpio -icvtB

Backup Multiple files/ dir. (with owners / group) For your reference only

(*) Not available in all systems, may need to download, see www.gnu.org

COMP111 Lecture a1 / Slide 5

Compress / Uncompress
Compression single file and available on most systems Compressed .z file smaller, file date preserved Actually compression ratio depend on data (text file and database files usually larger)
$ ls -l buyer* 22651778 Nov 16 18:45 buyer.dbf $ compress buyer.dbf $ ls -l buyer* 3603257 Nov 16 18:45 buyer.dbf.Z $ uncompress buyer.dbf.Z $ ls -l buyer* 22651778 Nov 16 18:45 buyer.dbf

COMP111 Lecture a1 / Slide 6

gzip Compression (1)

Backward compatible with compress, better compression ratio

Available from GNU www.gnu.org

$ gzip h gzip 0.7 (5 Jan 93) usage: gzip [-cdfhLrtvV19] [file ...] -c --stdout write on standard output, keep original files unchanged -d --decompress decompress -f --force force overwrite of output file and compress links -h --help give this help -L --license display software license -r --recurse recurse through directories -t --test test compressed file integrity -v --verbose verbose mode -V --version display version number -1 --fast compress faster -9 --best compress better file... files to (de)compress. If none given, use standard input.

COMP111 Lecture a1 / Slide 7

gzip Compression (2)

$ ls -l buyer* 22651778 Nov 16 18:45 buyer.dbf Note the file date is preserved $ gzip buyer.dbf gzip usually compresses more $ ls -l buyer* May be default to .z (in Sun) 3363713 Nov 16 18:45 buyer.dbf.gz $ gzip -l buyer* compressed uncompressed ratio uncompressed_name 3363713 22651778 85.1% buyer.dbf $ gzip -d buyer.dbf.gz gzip l available in newer versions (or gzip --decompress buyer.dbf.gz) (or gzip -d buyer.dbf) $ compress buyer* Also decompress old compress $ gzip -d buyer.dbf.Z format .Z files

COMP111 Lecture a1 / Slide 8

zip Compression (PC compatible)

Copyright (C) 1990-1996 Mark Adler, Richard B. Wales, Jean-loup Gailly Onno van der Linden and Kai Uwe Rommel. Type 'zip -L' for the software License. Zip 2.1 (April 27th 1996). Usage: zip [-options] [-b path] [-t mmddyy] [-n suffixes] [zipfile list] [-xi list] The default action is to add or replace zipfile entries from list, which can include the special name - to compress standard input. If zipfile and list are omitted, zip compresses stdin to stdout. -f freshen: only changed files -u update: only changed or new files -d delete entries in zipfile -m move into zipfile (delete files) -k force MSDOS (8+3) file names -g allow growing existing zipfile -r recurse into directories -j junk (don't record) directory names -0 store only -l convert LF to CR LF (-ll CR LF to LF) -1 compress faster -9 compress better -q quiet operation -v verbose operation/print version info -c add one-line comments -z add zipfile comment -b use "path" for temp file -t only do files after "mmddyy" -@ read names from stdin -o make zipfile as old as latest entry -x exclude the following names -i include only the following names -F fix zipfile (-FF try harder) -D do not add directory entries -A adjust self-extracting exe -J junk zip file prefix (unzipsfx) -T test zipfile integrity -X eXclude eXtra file attributes -y store symbolic links as the link instead of the referenced file -h show this help -n don't compress these suffixes

COMP111 Lecture a1 / Slide 9

zip (PC pkzip compatible)

Create compressed archives of files or directory trees
$ zip buyer buyer.dbf adding: buyer.dbf (deflated 85%) $ ls -al buyer* 22651778 Nov 16 18:45 buyer.dbf 3363835 Mar 1 23:48 buyer.zip $ zip -r /tmp/zipfile * adding: letter0 (stored 0%) adding: secret/ (stored 0%) adding: secret/letter1 (deflated adding: secret/letter2 (deflated
original file is kept unless m used

recursive compressing / backup of a directory tree

14%) 19%)

COMP111 Lecture a1 / Slide 10

unzip - PC pkunzip compatible (1)

UnZip 5.41 of 16 April 2000, by Info-ZIP. Maintained by C. Spieler. Send bug reports to the authors at Zip-Bugs@lists.wku.edu; see README for details. Usage: unzip [-Z] [-opts[modifiers]] file[.zip] [list] [-x xlist] [-d exdir] Default action is to extract files in list, except those in xlist, to exdir; file[.zip] may be a wildcard. -Z => ZipInfo mode ("unzip -Z" for usage). -p extract files to pipe, no messages -l list files (short format) -f freshen existing files, create none -t test compressed archive data -u update files, create if necessary -z display archive comment -x exclude files that follow (in xlist) -d extract files into exdir modifiers: -q quiet mode (-qq => quieter) -n never overwrite existing files -a auto-convert any text files -o overwrite files WITHOUT prompting -aa treat ALL files as text -j junk paths (do not make directories) -v be verbose/print version info -C match filenames case-insensitively -L make (some) names lowercase -X restore UID/GID info -V retain VMS version numbers -M pipe through "more" pager Examples (see unzip.doc for more info): unzip data1 -x joe => extract all files except joe from zipfile data1.zip unzip -p foo | more => send contents of foo.zip via pipe into program more unzip -fo foo ReadMe => quietly replace existing ReadMe if archive file newer

COMP111 Lecture a1 / Slide 11

unzip - PC pkunzip compatible (2)

Viewing contents of a zip file
$ unzip -v /tmp/zipfile
Archive: Length -----9 0 14 16 -----39 /tmp/zipfile.zip Method Size Ratio --------- ----Stored 9 0% Stored 0 0% Defl:N 12 14% Defl:N 13 19% ------ --34 13% Date ---03-01-01 03-01-01 03-01-01 03-01-01 Time ---23:13 23:14 23:14 23:14 CRC-32 -----590af8aa 00000000 b19b8197 f5c37f9c Name ---letter0 secret/ secret/letter1 secret/letter2 ------4 files

COMP111 Lecture a1 / Slide 12

unzip - PC pkunzip compatible (3)

Extracting all files from a zip file
unzip /tmp/zipfile Archive: /tmp/zipfile.zip extracting: letter0 creating: secret/ inflating: secret/letter1 inflating: secret/letter2
$ cd /tmp ;

Extracting selected files from a zip file

$ unzip zipfile "secret/letter*" Archive: zipfile.zip replace secret/letter1? [y]es, [n]o, [A]ll, [N]one, [r]ename: y inflating: secret/letter1 replace secret/letter2? [y]es, [n]o, [A]ll, [N]one, [r]ename: y inflating: secret/letter2

COMP111 Lecture a1 / Slide 13

tar archive no compression (1)

Traditional archive utility backup directory tree(s)
$ a a a a a tar cvf /tmp/tar1.tar . ./ 0K ./letter0 1K ./secret/ 0K ./secret/letter1 1K ./secret/letter2 1K $ a a a a tar cvf /tmp/tar2.tar * letter0 1K secret/ 0K secret/letter1 1K secret/letter2 1K

$ tar tvf /tmp/tar1.tar tar: blocksize = 10 drwxr-xr-x -rw-r--r-drwxr-xr-x -rw-r--r--rw-r--r-4110/1501 4110/1501 4110/1501 4110/1501 4110/1501 0 9 0 14 16

Replace output file with tape drive device file, e.g. /dev/st[0-7] (in Linux) /dev/rmt/[0-7]* (in Sun) /dev/rStp[0-7] (in SCO UNIX)

Mar Mar Mar Mar Mar

1 1 1 1 1

23:14 23:13 23:14 23:14 23:14

2001 2001 2001 2001 2001

./ ./letter0 ./secret/ ./secret/letter1 ./secret/letter2

COMP111 Lecture a1 / Slide 14

tar archive no compression (2)

Extracting directories and files from a tar file
$ cd /tmp ; tar xvf /tmp/tar1.tar tar: blocksize = 10 x ., 0 bytes, 0 tape blocks x ./letter0, 9 bytes, 1 tape blocks x ./secret, 0 bytes, 0 tape blocks x ./secret/letter1, 14 bytes, 1 tape blocks x ./secret/letter2, 16 bytes, 1 tape blocks tar: can't set time on .: Not owner $ tar Usage: tar {txruc}[vfbFXhiBEelmopwnq[0-7]] [-k size] [tapefile] [blocksize] [exclude-file] [-I includefile] files ...

COMP111 Lecture a1 / Slide 15

tar archive selective restore

$ tar xvf tar1.tar ./letter0 tar: blocksize = 10 x ./letter0, 9 bytes, 1 tape blocks $ tar xvf tar1.tar `tar tvf tar1.tar | grep secret` tar: blocksize = 10 The backquote sub-command gets a list of files to restore in the first pass tar: blocksize = 10 x ./secret, 0 bytes, 0 tape blocks x ./secret/letter1, 14 bytes, 1 tape blocks x ./secret/letter2, 16 bytes, 1 tape blocks

COMP111 Lecture a1 / Slide 16

tar archive with compression (1)

Use to set tar s output / input to stdout / stdin so that compress/gzip can process
$ tar cf - . | compress > /tmp/tar1.tar.Z $ uncompress < /tmp/tar1.tar.Z | tar tvf tar: blocksize = 10 drwxr-xr-x 4110/1501 -rw-r--r-- 4110/1501 drwxr-xr-x 4110/1501 -rw-r--r-- 4110/1501 -rw-r--r-- 4110/1501 0 9 0 14 16 Mar Mar Mar Mar Mar 1 1 1 1 1 23:14 23:13 23:14 23:14 23:14 2001 2001 2001 2001 2001 ./ ./letter0 ./secret/ ./secret/letter1 ./secret/letter2

$ tar xvf /tmp/tar1.tar.Z Cannot extract directly, need uncompress tar: tape blocksize error $ cd /tmp; uncompress < /tmp/tar1.tar.Z | tar xvf tar: blocksize = 10 gtar xvfz /tmp/tar1.tar.Z x ., 0 bytes, 0 tape blocks x ./letter0, 9 bytes, 1 tape blocks x ./secret, 0 bytes, 0 tape blocks x ./secret/letter1, 14 bytes, 1 tape blocks x ./secret/letter2, 16 bytes, 1 tape blocks tar: can't set time on .: Not owner

COMP111 Lecture a1 / Slide 17

tar archive with compression (2)

Similarly, we can use gzip instead of compress
$ tar cf - . | gzip -c > /tmp/tar2.tar.z $ gzip -c -d < /tmp/tar2.tar.z | tar tvf tar: blocksize = 10 drwxr-xr-x 4110/1501 -rw-r--r-- 4110/1501 drwxr-xr-x 4110/1501 -rw-r--r-- 4110/1501 -rw-r--r-- 4110/1501 0 9 0 14 16 Mar Mar Mar Mar Mar 1 1 1 1 1 23:14 23:13 23:14 23:14 23:14 2001 2001 2001 2001 2001 ./ ./letter0 ./secret/ ./secret/letter1 ./secret/letter2

$ tar xvf /tmp/tar2.tar.z gzip c: use standard input/output tar: tape blocksize error $ cd /tmp; gzip -c -d < /tmp/tar2.tar.z | tar xvf tar: blocksize = 10 x ., 0 bytes, 0 tape blocks gtar xvfz /tmp/tar2.tar.z x ./letter0, 9 bytes, 1 tape blocks x ./secret, 0 bytes, 0 tape blocks x ./secret/letter1, 14 bytes, 1 tape blocks x ./secret/letter2, 16 bytes, 1 tape blocks tar: can't set time on .: Not owner

COMP111 Lecture a1 / Slide 18

cpio - selected reference (1)

cpio backups with file owner and group cpio need a list of files to be backup from stdin and sends archive file to stdout
$ find . -print | cpio -ocvB > /tmp/cpio1
. letter0 secret secret/letter1 secret/letter2 10 blocks

Restoring files from cpio archives

$ cpio -icvtB < /tmp/cpio1
drwxr-xr-x -rw-r--r-drwxr-xr-x -rw-r--r--rw-r--r-10 blocks 3 1 2 1 1 kwchiu kwchiu kwchiu kwchiu kwchiu cs 0 Mar cs 9 Mar cs 0 Mar cs 14 Mar cs 16 Mar 1 1 1 1 1 23:14 23:13 23:14 23:14 23:14 2001, 2001, 2001, 2001, 2001, . letter0 secret secret/letter1 secret/letter2

COMP111 Lecture a1 / Slide 19

cpio - selected reference (2)

cpio backups with file owner and group
$ cpio -icvdB < /tmp/cpio1
. cpio -icvdBu (-u unconditional overwrite) cpio: Existing "letter0" same age or newer secret cpio: Existing "secret/letter1" same age or newer cpio: Existing "secret/letter2" same age or newer 10 blocks 3 error(s)

cpio can also be used with gzip / compress

$ find . -print | cpio -ocvB | gzip -d > /tmp/cpio2.z $ gzip -c -d < /tmp/cpio2.z | cpio -icvdB $ gzip -c -d < /tmp/cpio2.z | cpio -icvtB

COMP111 Lecture a1 / Slide 20

Summary
Archive Restore List Usage Notes

compress
(*) gzip

uncompress
gzip -d gzip -l

Compress single file

Backward compatible with compress Compress (better ratio) single file

(*) zip (-r)

tar cvf

unzip
tar xvf

unzip -v
tar cvf

DOS pkzip / Winzip compatible Compress/backup multiple files/ dir.

Backup multiple files/ dir. (cf. gtar (*))

cpio -ocvB cpio -icvdB cpio -icvtB

Backup Multiple files/ dir. (with owners / group) For your reference only

(*) Not available in all systems, may need to download, see www.gnu.org