H

ow ephemeral is content on the internet? Can a library rea-

If you want to sonably expect to collect each new version of its website or
blog as it would collect every issue of a magazine or journal?
maintain your web Large-scale projects such as the Internet Archive (www.ar
chive.org) send out crawlers to gather snapshots of much of the
archive into the web. This massive collection of archived websites may include content of
interest to your patrons. But if you want to control exactly when and what
future, you will need is archived, relying on someone else to do the archiving isn’t ideal.
Though large-scale web archiving and preservation are outside the mis-
to think about sion of most libraries, it is possible for even the smallest operations to main-
tain an archive of a group of sites. This article outlines a simple workflow
long-term for Macs that uses free software and requires only the most basic techni-
cal expertise. There is no programming involved, and you won’t have to
touch a command-line interface. This is web archiving for the rest of us.
preservation. We developed this process in response to an assignment at the Sim-
mons College Graduate School of Library and Information Science. At
Simmons, Katharine is a student and an editorial fellow, and Nick is a
student while he works as a library assistant in preservation services
at the Massachusetts Institute of Technology (MIT) Libraries. We have
put this process to the test on relatively static webpages, blogs, and even
the social networking site Twitter.

12 | SEPTEMBER 2009 » www.infotoday.com

 by
Katharine Dunn and
Nick Szydlowski
 COMPUTERS IN LIBRARIES
web archiving for the rest of us
When you’re learning to use SiteSucker, it is
best to start with a target webpage that
consists primarily of plain HTML.
Workflow
There are just three things you need
to do to create an archive of websites.
(Preservation is another matter; we’ll dis-
cuss this briefly at the end of the article.)
1. Harvesting: First you must
acquire, or harvest, the content
you are collecting. Because many
websites change frequently, you
will need to reacquire a current
version of the site at some desig-
nated interval.
2. Version Control: Once you
have harvested multiple versions
of a site, you will need to keep
track of the versions so you can de-
termine which iterations are dif-
ferent enough from one another to
be worth keeping.
3. Presentation: The reason to
archive sites is to use them again,
right? How you choose to make
your new web archive available to
your patrons will depend on what
resources you have at your
disposal. But it is possible to
view the files on a computer
(PC or Mac), burn them to a
CD, or even place them on a
server for remote viewing.
Harvesting A SiteSucker screen
In order to efficiently acquire
entire sites, we chose to use a piece of
software called SiteSucker. This soft-
ware was designed to rapidly download
entire websites and place the files on
your hard drive in a folder structure
that mirrors the way the site was
14 | SEPTEMBER 2009 » www.infotoday.com
arranged on its server. SiteSucker is
donation-ware and can be down-
loaded from www.sitesucker.us. PC
and Linux users will find that the
free software HTTrack (available at
www.httrack.com) offers similar
functionality.
To download a site using Site-
Sucker, simply type the URL of the site
into the box marked “Web URL” and
press the “Download” button. Site-
Sucker will follow the links on each
page, downloading every file that a
user can access to a user-designated
set of folders on your hard drive.
Clicking on the “Settings” icon gives
you much greater control over the
downloaded files. For example, it is
possible to specify which file exten-
sions the program should download
and which it should ignore. Use the
settings to limit or expand the number
of files downloaded and to ensure that
you get the files you need. It is possi-
ble to save your settings so that each
time you download a page, SiteSucker
performs the same tasks.
When you’re learning to use Site-
Sucker, it is best to start with a target
webpage that consists primarily of
plain HTML rather than a site where
content is held in a database and has
pages generated by scripts. If you have
trouble determining the difference,
look for pages that end with the file ex-
tensions .htm or .html—these are more
likely to be plain HTML. SiteSucker
cannot follow links that appear within
JavaScript, so some types of sites will
not render correctly when harvested
with this program. We discuss the de-
tails of using SiteSucker on database-
backed sites later.
Version Control
In the software development com-
munity, the term “version control”
refers to the process of managing the
updates of files and documents, such
as software code, that may be edited or
changed many times by multiple users.
Version control software such as CVS
or Subversion makes it possible to re-
turn to any past version of a file. In
essence, version control gives you in-
tellectual control over the content you
collect—and if you have born-digital
content, it can be essential when per-
forming traditional library and archive
tasks such as cataloging, appraisal,
and providing access. Version control
helps us reach one of the end goals of
web archiving: making past versions
available to users.
When we first started working on
this project, we attempted to use Sub-
version, a popular and free version con-
trol package, to track different itera-
tions of the sites we downloaded.
However, as nonprogrammers
unaccustomed to using a com-
mand-line interface, we strug-
gled to produce usable results
with Subversion. As a tool cre-
ated for the software develop-
ment community, it is perhaps
best suited for those with some
programming skills.
In response to these difficulties, we
developed a two-tiered version control
method using programs created by Ap-
ple. The first of these is Time Machine,
which is preinstalled as part of the
Mac OS 10.5 (Leopard) or higher. The
 COMPUTERS IN LIBRARIES
web archiving for the rest of us

second, FileMerge, is part of the free chine may immediately start a backup. After collecting a number of ver-
Apple Xcode developer package. Both If not, click on the clock icon in the sions of your website, you may want
programs feature intuitive graphic in- menu bar on the upper right of your to compare these versions and make
terfaces, and neither requires any spe- screen and scroll to “Back Up Now.” appraisal decisions about which are
cial skills. Using these programs, it is You can continue working while the worth keeping and which you can put
possible to maintain control of the dif- computer is performing a backup. It is in the trash. To retrieve older versions
ferent versions of the websites you’re wise to make sure your external hard of your website from Time Machine,
tracking and to quickly appraise new drive has plenty of space since when make sure your external hard drive is
versions to determine if they are dif- the disk becomes full, Time Machine plugged in and turned on. Click the
ferent from previous iterations. will begin to erase the oldest versions icon and press “Enter Time Machine.”
Step 1: Managing backups with to make room for new ones. A Finder window (which shows your
Time Machine. As mentioned, Time
Machine comes preinstalled on Macs
with operating system versions 10.5 or
higher. Most Macs purchased since late
2007 should come with this program,
which creates incremental backups,
meaning that each new backup does
not override previous versions. This
feature allows you to track down and
restore files that, for example, you may
have accidentally deleted from your
hard drive. The program can be set up
to back up your entire hard drive, mul-
tiple files, or a single file onto an ex-
ternal drive.
It is possible to achieve the same
end results without using Time Ma-
chine, but using Time Machine has two
significant benefits:

• You will save on disk space

because all archived versions of
files will be on an external drive
rather than on your computer.
Time Machine will manage the
versions efficiently rather than
using disk space to store multiple
copies of the same file.

• Because an external drive holds

incremental backups, you can
delay appraisal decisions. This
will allow you to batch these
decisions, creating a more flexible
and efficient workflow.

To begin, you should run a backup

using Time Machine after each web
harvest. First, connect the external
hard drive you will be using to store
your backed-up versions. Time Ma-
Here are two screen shots of the Time Machine in action. On April 6, we entered the Time Machine and searched “sahughes,”
the folder name of our website. Pressing the back arrow on the lower right-hand side took us to the files as they existed the
first time we backed them up, on March 21 (first screen shot). The second screen shot is of the files on April 6 (“Today (Now)”).

16 | SEPTEMBER 2009 » www.infotoday.com


files and folders) will drop down in
front of an identical stack of windows
on an “outer space” backdrop.
Enter the name or partial name of
your file or folder in the spotlight
search box on the upper right of the
Finder window. Time Machine searches
for the files and presents the most re-
cent version in the Finder window at
the front. To go to earlier versions,
Here are two screen shots of FileMerge. We asked the program to compare the April 6 and April 2 versions of the sahughes
files. The program lists individual files and notes differences between the two versions. Here, we’ve highlighted the file
“travel.html,” which has four changes. We then double-clicked on travel.html, and a window popped up, showing the newer
and the older versions of the file in columns next to each other.
press the back arrow on the lower
right of the screen. You can see every
version saved along with the date it
was saved. Select the versions you’re
interested in and copy and paste each
one to the desktop. Rename each
folder, adding the date of the files at
the end, e.g., “Library_Website_June29.”
Do not press the “Restore” button on
the bottom right of the screen. If you
COMPUTERS IN LIBRARIES
web archiving for the rest of us
do so, the version of the files that you
are looking at will replace the latest ver-
sion, and you may lose some files.
The folders containing the files are
now clearly labeled and ready to be
compared. Time Machine is limited in
this way; it holds versions but cannot
show you what changes have been
made. To do this you need the program
FileMerge.
Step 2: Comparing versions of
the site with FileMerge. FileMerge
is included in Apple’s Xcode developer
package, which is freely available at
http://developer.apple.com/tools/xcode.
The program compares two similar
folders and shows any differences be-
tween them. When you run the appli-
cation, it will prompt you for two fold-
ers or files, labeled “right” and “left.”
Begin by selecting the first and second
versions of the site you have archived.
When you press the “Compare” button,
FileMerge will display a folder struc-
ture, showing all files that appear in
either folder. Files that are identical in
both folders will appear in gray. Files
that are only present in one folder or
the other, or which are not exactly the
same in each folder, will appear in
black. The check boxes at the upper
right of the window allow you to
choose which categories of file will be
displayed.
You may wish to make an appraisal
decision based simply on how many
files have been added, removed, or
changed from one version of the site to
another. Depending on the goals of your
archive, you may decide that a new ver-
sion of the site only merits retention if
it includes more than 10 new files, or
that any change at all represents a new
version that should be kept.
If you are familiar with HTML, you
can also use FileMerge to view more
specifically what has changed about a
page without having to read the entire
contents of the page. To do this, select
the page in FileMerge and choose
“Comparison” from the drop-down menu
“View.” FileMerge will display the code
www.infotoday.com « SEPTEMBER 2009 | 17
 COMPUTERS IN LIBRARIES
web archiving for the rest of us
of the two versions side by side, with
each change highlighted and enumer-
ated. Using this information, you may
find it easier to make a precise deter-
mination about whether changes are
significant or minor. Of course, what
constitutes a significant change will
depend on the nature of your project.
If you determine that a version of the
site is not different enough to keep,
simply delete it from your computer
and/or external hard drive.
Presentation
Once you have decided to retain a
particular version of a site, you’re
ready to make that version available
to your patrons. From a technical
standpoint, you can serve the files by
any method you would use for other
HTML files: You can host them on a lo-
cal computer or network, lend them
out on a CD-R, or even place them on
a public web server. Using any of these
methods, it is important to direct the
user first to the index file (either in-
dex.html, index.htm, or index.php).
This can be achieved by written in-
structions or by linking directly to the
file from a menu. Care should be taken
to avoid violating the copyright of con-
tent owners. Though the law in this
area remains somewhat unsettled, a
useful and brief guide is the Oakland
Archive Policy (http://www2.sims.ber
keley.edu/research/conferences/aps/
removal-policy.html), which has been
adopted by the Internet Archive.
Notes on Blogs and Other
Database-Backed Sites
SiteSucker, the software we’ve used
to harvest websites, was primarily de-
signed to work on sites based on plain
HTML. When you harvest a plain
HTML site, the end product should
mirror what was originally on the web
server: the same files in the same folder
structure. However, more and more
sites these days are designed using
18 | SEPTEMBER 2009 » www.infotoday.com
FileMerge compares two similar folders and
shows any differences between them.
software that holds web content in a
database and generates pages dynam-
ically, based on the user’s request. This
includes sites created using either blog-
ging software or a content management
system (CMS) such as Joomla.
In order to mirror these sites ex-
actly as they exist on their servers,
SiteSucker would have to download
the database software from the server
and run it on your computer. For many
reasons, this is not possible. What
SiteSucker will do instead when pre-
sented with this type of site is to fol-
low every link on every page. In data-
base-backed sites, this can produce
thousands of possibilities because
each piece of content can often be dis-
played in a variety of contexts.
SiteSucker’s settings can help limit
your archive to a reasonable number
of files, either by setting a hard limit
on the number of files downloaded or
by limiting the number of links the
program will follow in succession, us-
ing the levels setting. Limiting levels
to two will return only the homepage
and files linked directly from it.
Sites created on many blogging and
CMS platforms do not render correctly
when downloaded by SiteSucker. This
is often because the formatting infor-
mation for these sites is contained in
scripts that SiteSucker cannot harvest
or interpret. Whether a particular
platform will work with SiteSucker
depends on the details of the software
itself. Fortunately, some of the most
popular blogging platforms, including
WordPress and Blogger, rendered cor-
rectly in our tests. For blogs created in
other software, we were able to archive
the content, but the formatting infor-
mation was not retained. SiteSucker
can also be used to harvest the main
page of a Twitter account, though it
cannot reach archived posts.
Conclusion
Archiving websites can be ex-
tremely useful, but simply owning the
files doesn’t guarantee they will con-
tinue to be usable. If you want to
maintain your web archive into the fu-
ture, you will need to think about
long-term preservation. This is a com-
plex issue, even for relatively stable
file formats such as HTML. But it can
be simplified by timely and thought-
ful action. Two great sources of infor-
mation on the topic are the Digital
Curation Centre in the U.K. (www.d
cc.ac.uk) and Australia’s PADI (Pre-
serving Access to Digital Information;
www.nla.gov.au/padi). An excellent
first step is to maintain consistent
and detailed metadata about the files
you are archiving. Simply knowing
the format and the creation date of a
file may make the difference between
a usable file and an unusable one, 20,
50, or 100 years from now.
Katharine Dunn (katharine.dunn
@simmons.edu) is a graduate student
in library and information science at
Simmons College in Boston. Dunn, who
is also a freelance magazine writer,
works as the school’s editorial fellow.
Nick Szydlowski is a library assistant
in preservation services at the MIT Li-
braries. He is currently pursuing an
M.L.S. from the Simmons College
Graduate School of Library Science. He
can be reached at nick_s@mit.edu.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.