TERM PAPER

OF
INT-301: WEB PROGRAMMING

Topic: SEARCH ENGINE

Submitted by:

AVINASH MANHAS Submitted to:

Roll.no- RE2801B46 Mr.SHAKUN GARG

Reg.no- 10809450

Course- B Tech-M.Tech( IT)

 search engine -- the first such detailed public
Search Engine description we know of to date.
Apart from the problems of scaling
WHAT ARE SEARCH ENGINES? traditional search techniques to data of this
magnitude, there are new technical
Search engines are huge databases of web
page files that have been assembled challenges involved with using the
automatically by machine. additional information present in hypertext
to produce better search results. This paper
There are two types of search engines: addresses this question of how to build a
practical large-scale system which can
1. Individual. Individual search
exploit the additional information present in
engines compile their own
searchable databases on the web. hypertext. Also we look at the problem of
2. Meta. Meta searchers do not how to effectively deal with uncontrolled
compile databases. Instead, they hypertext collections where anyone can
search the databases of multiple sets publish anything they want.
of individual engines simultaneously
Keywords: World Wide Web, Search
Abstract Engines, Information Retrieval, PageRank,
In this , we present Google, a prototype of a Google
large-scale search engine which makes
heavy use of the structure present in
hypertext. Google is designed to crawl and
index the Web efficiently and produce much
more satisfying search results than existing
systems. The prototype with a full text and
hyperlink database of at least 24 million
pages is available at
http://google.stanford.edu/
To engineer a search engine is a
challenging task. Search engines index tens
to hundreds of millions of web pages
involving a comparable number of distinct
terms. They answer tens of millions of
queries every day. Despite the importance of
large-scale search engines on the web, very
little academic research has been done on
them. Furthermore, due to rapid advance in
technology and web proliferation, creating a
web search engine today is very different
from three years ago. This paper provides an
in-depth description of our large-scale web
 a. Example - Search Terms
Which Yield Too Many
CONTENTS: Matches

b. Always Search As
1. Introduction
Specifically As Possible
Web Search Engines -- Scaling Up: 1994 – c. Think About Search Terms
2000
a. Google: Scaling with the
Web

2. History

8. Results and Performance

3. Design Goals a. Storage Requirements
b. System Performance
Improved Search Quality c. Search Performance
a. Academic Search Engine 9. Conclusions
Research a. Future Work
b. High Quality Search
c. Scalable Architecture
d. A Research Tool
4. System Features 10. References

a. Page Rank: Bringing Order to

the Web

b. Description of Page Rank

Calculation

c. Intuitive Justification

d. Other Features

How search engines work

5. PROS AND CONS OF SEARCH

ENGINES

6. Search Engine Features

7. Which is the Best Search Engine?


Introduction
When we want to find something on the web
we look to a search engine, such as those in
Figure 1. Sites like Google, MSN and
Yahoo! let you search for web sites that
contain information pertinent to topics of
interest to you. Potential visitors looking
for your site are going to do the same
thing. This makes it imperative that your
site get ranked high enough for important
keywords that visitors can find it. Knowing
what keywords are important means
knowing what visitors are looking for when
they find your site.
Figure 1. Search Engines are the most
common tool for promoting a web site
The web creates new challenges for
information retrieval. The amount of
information on the web is growing rapidly,
as well as the number of new users
inexperienced in the art of web research.
People are likely to surf the web using its
link graph, often starting with high quality
human maintained indices such as Yahoo! or
with search engines. Human maintained lists
cover popular topics effectively but are
subjective, expensive to build and maintain,
slow to improve, and cannot cover all
esoteric topics. Automated search engines
that rely on keyword matching usually
return too many low quality matches. To
make matters worse, some advertisers
attempt to gain people's attention by taking
measures meant to mislead automated
search engines. We have built a large-scale
search engine which addresses many of the
problems of existing systems. It makes
especially heavy use of the additional
structure present in hypertext to provide
much higher quality search results. We
chose our system name, Google, because it
is a common spelling of googol, or 10100 and
fits well with our goal of building very
large-scale search engines.
Web Search Engines -- Scaling Up:
1994 - 2000
Search engine technology has had to scale
dramatically to keep up with the growth of
the web. In 1994, one of the first web search
engines, the World Wide Web Worm
(WWWW) [McBryan 94] had an index of
110,000 web pages and web accessible
documents. As of November, 1997, the top
search engines claim to index from 2 million
(WebCrawler) to 100 million web
documents (from Search Engine Watch). It
is foreseeable that by the year 2000, a
comprehensive index of the Web will
contain over a billion documents. At the
same time, the number of queries search
engines handle has grown incredibly too. In
March and April 1994, the World Wide Web
Worm received an average of about 1500
queries per day. In November 1997,
Altavista claimed it handled roughly 20
million queries per day. With the increasing
number of users on the web, and automated
systems which query search engines, it is
likely that top search engines will handle
hundreds of millions of queries per day by
the year 2000. The goal of our system is to
address many of the problems, both in
quality and scalability, introduced by scaling
search engine technology to such
extraordinary numbers.
Google: Scaling with the Web
Creating a search engine which scales even
to today's web presents many challenges.
Fast crawling technology is needed to gather
the web documents and keep them up to
date. Storage space must be used efficiently
to store indices and, optionally, the
documents themselves. The indexing system
must process hundreds of gigabytes of data
efficiently. Queries must be handled
quickly, at a rate of hundreds to thousands
per second.
These tasks are becoming increasingly
difficult as the Web grows. However,
hardware performance and cost have
improved dramatically to partially offset the
difficulty. There are, however, several
notable exceptions to this progress such as
disk seek time and operating system
robustness. In designing Google, we have
considered both the rate of growth of the
Web and technological changes. Google is
designed to scale well to extremely large
data sets. It makes efficient use of storage
space to store the index. Its data structures
are optimized for fast and efficient access.
Further, we expect that the cost to index and
store text or HTML will eventually decline
relative to the amount that will be available.
This will result in favorable scaling
properties for centralized systems like
Google.
History
During the early development of the web,
there was a list of webservers edited by Tim
Berners-Lee and hosted on the CERN
webserver. One historical snapshot from
1992 remains. As more webservers went
online the central list could not keep up. On
the NCSA site new servers were announced
under the title "What's New!"
The very first tool used for searching on the
Internet was Archie. The name stands for
"archive" without the "v." It was created in
1990 by Alan Emtage, Bill Heelan and J.
Peter Deutsch, computer science students at
McGill University in Montreal. The program
downloaded the directory listings of all the
files located on public anonymous FTP (File
Transfer Protocol) sites, creating a
searchable database of file names; however,
Archie did not index the contents of these
sites since the amount of data was so limited
it could be readily searched manually.
The rise of Gopher (created in 1991 by Mark
McCahill at the University of Minnesota)
led to two new search programs, Veronica
and Jughead. Like Archie, they searched the
file names and titles stored in Gopher index
systems. Veronica (Very Easy Rodent-
Oriented Net-wide Index to Computerized
Archives) provided a keyword search of
most Gopher menu titles in the entire
Gopher listings. Jughead (Jonzy's Universal
Gopher Hierarchy Excavation And Display)
was a tool for obtaining menu information
from specific Gopher servers. While the
name of the search engine "Archie" was not
a reference to the Archie comic book series,
"Veronica" and "Jughead" are characters in
the series, thus referencing their
predecessor.
In the summer of 1993, no search engine
existed yet for the web, though numerous
specialized catalogues were maintained by
hand. Oscar Nierstrasz at the University of
Geneva wrote a series of Perl scripts that
would periodically mirror these pages and
rewrite them into a standard format which
formed the basis for W3Catalog, the web's
first primitive search engine, released on
September 2, 1993.
In June 1993, Matthew Gray, then at MIT,
produced what was probably the first web
robot, the Perl-based World Wide Web
Wanderer, and used it to generate an index
called 'Wandex'. The purpose of the
Wanderer was to measure the size of the
World Wide Web, which it did until late
1995. The web's second search engine
Aliweb appeared in November 1993. Aliweb
did not use a web robot, but instead
depended on being notified by website
administrators of the existence at each site
of an index file in a particular format.
JumpStation (released in December 1993)
used a web robot to find web pages and to
build its index, and used a web form as the
interface to its query program. It was thus
the first WWW resource-discovery tool to
combine the three essential features of a web
search engine (crawling, indexing, and
searching) as described below. Because of
the limited resources available on the
platform on which it ran, its indexing and
hence searching were limited to the titles
and headings found in the web pages the
crawler encountered.
One of the first "full text" crawler-based
search engines was WebCrawler, which
came out in 1994. Unlike its predecessors, it
let users search for any word in any
webpage, which has become the standard for
all major search engines since. It was also
the first one to be widely known by the
public. Also in 1994, Lycos (which started
at Carnegie Mellon University) was
launched and became a major commercial
endeavor.
Soon after, many search engines appeared
and vied for popularity. These included
Magellan, Excite, Infoseek, Inktomi,
Northern Light, and AltaVista. Yahoo! was
among the most popular ways for people to
find web pages of interest, but its search
function operated on its web directory,
rather than full-text copies of web pages.
Information seekers could also browse the
directory instead of doing a keyword-based
search.
In 1996, Netscape was looking to give a
single search engine an exclusive deal to be
their featured search engine. There was so
much interest that instead a deal was struck
with Netscape by five of the major search
engines, where for $5Million per year each
search engine would be in a rotation on the
Netscape search engine page. The five
engines were Yahoo!, Magellan, Lycos,
Infoseek, and Excite.
Search engines were also known as some of
the brightest stars in the Internet investing
frenzy that occurred in the late 1990s.
Several companies entered the market
spectacularly, receiving record gains during
their initial public offerings. Some have
taken down their public search engine, and
are marketing enterprise-only editions, such
as Northern Light. Many search engine
companies were caught up in the dot-com
bubble, a speculation-driven market boom
that peaked in 1999 and ended in 2001.
Around 2000, the Google search engine rose
to prominence. company achieved better
results for many searches with an innovation
called PageRank. This iterative algorithm
ranks web pages based on the number and
PageRank of other web sites and pages that
link there, on the premise that good or
desirable pages are linked to more than
others. Google also maintained a minimalist
interface to its search engine. In contrast,
many of its competitors embedded a search
engine in a web portal.
By 2000, Yahoo was providing search
services based on Inktomi's search engine.
Yahoo! acquired Inktomi in 2002, and
Overture (which owned AlltheWeb and
AltaVista) in 2003. Yahoo! switched to
Google's search engine until 2004, when it
launched its own search engine based on the
combined technologies of its acquisitions.
Microsoft first launched MSN Search in the
fall of 1998 using search results from
Inktomi. In early 1999 the site began to
display listings from Looksmart blended
with results from Inktomi except for a short
time in 1999 when results from AltaVista
were used instead. In 2004, Microsoft began
a transition to its own search technology,
powered by its own web crawler (called
msnbot).
Microsoft's rebranded search engine, Bing,
was launched on June 1, 2009. On July 29,
2009, Yahoo! and Microsoft finalized a deal
in which Yahoo! Search would be powered
by Microsoft Bing technology.
Design Goals
Improved Search Quality
Our main goal is to improve the quality of
web search engines. In 1994, some people
believed that a complete search index would
make it possible to find anything easily.
According to Best of the Web 1994 --
Navigators, "The best navigation service
should make it easy to find almost anything
on the Web (once all the data is entered)."
However, the Web of 1997 is quite different.
Anyone who has used a search engine
recently, can readily testify that the
completeness of the index is not the only
factor in the quality of search results. "Junk
results" often wash out any results that a
user is interested in. In fact, as of November
1997, only one of the top four commercial
search engines finds itself (returns its own
search page in response to its name in the
top ten results). One of the main causes of
this problem is that the number of
documents in the indices has been increasing
by many orders of magnitude, but the user's
ability to look at documents has not. People
are still only willing to look at the first few
tens of results. Because of this, as the
collection size grows, we need tools that
have very high precision (number of
relevant documents returned, say in the top
tens of results). Indeed, we want our notion
of "relevant" to only include the very best
documents since there may be tens of
thousands of slightly relevant documents.
This very high precision is important even at
the expense of recall (the total number of
relevant documents the system is able to
return). There is quite a bit of recent
optimism that the use of more hypertextual
information can help improve search and
other applications [Marchiori 97] [Spertus
97] [Weiss 96] [Kleinberg 98]. In particular,
link structure and link text provide a lot of
information for making relevance judgments
and quality filtering. Google makes use of
both link structure and anchor text.
Academic Search Engine Research
Aside from tremendous growth, the Web has
also become increasingly commercial over
time. In 1993, 1.5% of web servers were
on .com domains. This number grew to over
60% in 1997. At the same time, search
engines have migrated from the academic
domain to the commercial. Up until now
most search engine development has gone
on at companies with little publication of
technical details. This causes search engine
technology to remain largely a black art and
to be advertising oriented . With Google,
we have a strong goal to push more
development and understanding into the
academic realm.
Another important design goal was to build
systems that reasonable numbers of people
can actually use. Usage was important to us
because we think some of the most
interesting research will involve leveraging
the vast amount of usage data that is
available from modern web systems. For
example, there are many tens of millions of
searches performed every day. However, it
is very difficult to get this data, mainly
because it is considered commercially
valuable.
Our final design goal was to build an
architecture that can support novel research
activities on large-scale web data. To
support novel research uses, Google stores
all of the actual documents it crawls in
compressed form. One of our main goals in
designing Google was to set up an
environment where other researchers can
come in quickly, process large chunks of the
web, and produce interesting results that
would have been very difficult to produce
otherwise. In the short time the system has
been up, there have already been several
papers using databases generated by Google,
and many others are underway. Another
goal we have is to set up a Spacelab-like
environment where researchers or even
students can propose and do interesting
experiments on our large-scale web data.
System Features
The Google search engine has two important
features that help it produce high precision
results. First, it makes use of the link
structure of the Web to calculate a quality
ranking for each web page. This ranking is
called PageRank and is described in detail in
[Page 98]. Second, Google utilizes link to
improve search results.
Page Rank: Bringing Order to the
Web
The citation (link) graph of the web is an
important resource that has largely gone
unused in existing web search engines. We
have created maps containing as many as
518 million of these hyperlinks, a significant
sample of the total. These maps allow rapid
calculation of a web page's "Page Rank", an
objective measure of its citation importance
that corresponds well with people's
subjective idea of importance. Because of
this correspondence, Page Rank is an
excellent way to prioritize the results of web
keyword searches. For most popular
subjects, a simple text matching search that
is restricted to web page titles performs
admirably when Page Rank prioritizes the
results (demo available at
google.stanford.edu). For the type of full
text searches in the main Google system,
Page Rank also helps a great deal.
Description of Page Rank
Calculation
Academic citation literature has been
applied to the web, largely by counting
citations or backlinks to a given page. This
gives some approximation of a page's
importance or quality. PageRank extends
this idea by not counting links from all
pages equally, and by normalizing by the
number of links on a page. PageRank is
defined as follows:
We assume page A has pages T1...Tn which
point to it (i.e., are citations). The parameter
d is a damping factor which can be set
between 0 and 1. We usually set d to 0.85.
There are more details about d in the next
section. Also C(A) is defined as the number
of links going out of page A. The PageRank
of a page A is given as follows:
PR(A) = (1-d) + d (PR(T1)/C(T1) + ... +
PR(Tn)/C(Tn))
Note that the PageRanks form a probability
distribution over web pages, so the sum of
all web pages' PageRanks will be one.
PageRank or PR(A) can be calculated using
a simple iterative algorithm, and
corresponds to the principal eigenvector of
the normalized link matrix of the web. Also,
a PageRank for 26 million web pages can be
computed in a few hours on a medium size
workstation. There are many other details
which are beyond the scope of this paper.
Intuitive Justification
PageRank can be thought of as a model of
user behavior. We assume there is a
"random surfer" who is given a web page at
random and keeps clicking on links, never
hitting "back" but eventually gets bored and
starts on another random page. The
probability that the random surfer visits a
page is its PageRank. And, the d damping
factor is the probability at each page the
"random surfer" will get bored and request
another random page. One important
variation is to only add the damping factor d
to a single page, or a group of pages. This
allows for personalization and can make it
nearly impossible to deliberately mislead the
system in order to get a higher ranking. We
have several other extensions to PageRank.
Another intuitive justification is that a page
can have a high PageRank if there are many
pages that point to it, or if there are some
pages that point to it and have a high
PageRank. Intuitively, pages that are well
cited from many places around the web are
worth looking at. Also, pages that have
perhaps only one citation from something
like the Yahoo! homepage are also generally
worth looking at. If a page was not high
quality, or was a broken link, it is quite
likely that Yahoo's homepage would not link
to it. PageRank handles both these cases and
everything in between by recursively
propagating weights through the link
structure of the web.
Other Features
Aside from PageRank and the use of anchor
text, Google has several other features. First,
it has location information for all hits and so
it makes extensive use of proximity in
search. Second, Google keeps track of some
visual presentation details such as font size
of words. Words in a larger or bolder font
are weighted higher than other words. Third,
full raw HTML of pages is available in a
repository.
How search engines work
A search engine operates, in the following
order
1. Web crawling
2. Indexing
3. Searching
Web search engines work by storing
information about many web pages, which
they retrieve from the html itself. These
pages are retrieved by a Web crawler
(sometimes also known as a spider) — an
automated Web browser which follows
every link on the site. Exclusions can be
made by the use of robots.txt. The contents
of each page are then analyzed to determine
how it should be indexed (for example,
words are extracted from the titles, headings,
or special fields called meta tags). Data
about web pages are stored in an index
database for use in later queries. A query
can be a single word. The purpose of an
index is to allow information to be found as
quickly as possible. Some search engines,
such as Google, store all or part of the
source page (referred to as a cache) as well
as information about the web pages, whereas
others, such as AltaVista, store every word
of every page they find. This cached page
always holds the actual search text since it is
the one that was actually indexed, so it can
be very useful when the content of the
current page has been updated and the
search terms are no longer in it. This
problem might be considered to be a mild
form of linkrot, and Google's handling of it
increases usability by satisfying user
expectations that the search terms will be on
the returned webpage. This satisfies the
principle of least astonishment since the user
normally expects the search terms to be on
the returned pages. Increased search
relevance makes these cached pages very
useful, even beyond the fact that they may
contain data that may no longer be available
elsewhere.
When a user enters a query into a search
engine (typically by using key words), the
engine examines its index and provides a
listing of best-matching web pages
according to its criteria, usually with a short
summary containing the document's title and
sometimes parts of the text. The index is
built from the information stored with the
data and the method by which the
information is indexed. Unfortunately, there
are currently no known public search
engines that allow documents to be searched
by date. Most search engines support the use
of the boolean operators AND, OR and NOT
to further specify the search query. Boolean
operators are for literal searches that allow
the user to refine and extend the terms of the
search. The engine looks for the words or
phrases exactly as entered. Some search
engines provide an advanced feature called
proximity search which allows users to
define the distance between keywords.
There is also concept-based searching where
the research involves using statistical
analysis on pages containing the words or
phrases you search for. As well, natural
language queries allow the user to type a
question in the same form one would ask it
to a human. A site like this would be
ask.com.
The usefulness of a search engine depends
on the relevance of the result set it gives
back. While there may be millions of web
pages that include a particular word or
phrase, some pages may be more relevant,
popular, or authoritative than others. Most
search engines employ methods to rank the
results to provide the "best" results first.
How a search engine decides which pages
are the best matches, and what order the
results should be shown in, varies widely
from one engine to another. The methods
also change over time as Internet usage
changes and new techniques evolve. There
are two main types of search engine that
have evolved: one is a system of predefined
and hierarchically ordered keywords that
humans have programmed extensively. The
other is a system that generates an "inverted
index" by analyzing texts it locates. This
second form relies much more heavily on
the computer itself to do the bulk of the
work.
Most Web search engines are commercial
ventures supported by advertising revenue
and, as a result, some employ the practice of
allowing advertisers to pay money to have
their listings ranked higher in search results.
Those search engines which do not accept
money for their search engine results make
money by running search related ads
alongside the regular search engine results.
The search engines make money every time
someone clicks on one of these ads.
PROS AND CONS OF SEARCH
ENGINES
PROS:
Search engines provide access to a fairly
large portion of the publicly available pages
on the Web, which itself is growing
exponentially.
Search engines are the best means devised
yet for searching the web. Stranded in the
middle of this global electronic library of
information without either a card catalog or
any recognizable structure, how else are you
going to find what you're looking for?
CONS:
On the down side, the sheer number of
words indexed by search engines increases
the likelihood that they will return hundreds
of thousands of responses to simple search
requests. Remember, they will return
lengthy documents in which your keyword
appears only once. Additionally, many of
these responses will be irrelevant to your
search.
ARE SEARCH ENGINES ALL THE
SAME?
Search engines use selected software
programs to search their indexes for
matching keywords and phrases, presenting
their findings to you in some kind of
relevance ranking. Although software
programs may be similar, no two search
engines are exactly the same in terms of
size, speed and content; no two search
engines use exactly the same ranking
schemes, and not every search engine offers
you exactly the same search options.
Therefore, your search is going to be
different on every engine you use. The
difference may not be a lot, but it could be
significant. Recent estimates put search
engine overlap at approximately 60 percent
and unique content at around 40 percent.
WHEN DO WE USE SEARCH
ENGINES?
Search engines are best at finding unique
keywords, phrases, quotes, and information
buried in the full-text of web pages. Because
they index word by word, search engines are
also useful in retrieving tons of documents.
If you want a wide range of responses to
specific queries, use a search engine.
NOTE: Today, the line between
search engines and subject
directoriesis blurring. Search engines
no longer limit themselves to a
search mechanism alone. Across the
Web, they are partnering with
subject directories, or creating their
own directories, and returning results
 gathered from a variety of other
guides and services as well.
Search Engine Features
Web location services typically specialize in
one of the following: their search tools (how
you specify a search and how the results are
presented), the size of their database, or their
catalog service. Most engines deliver too
many matches in a casual search, so the
overriding factor in their usefulness is the
quality of their search tools. Every search
engine I used had a nice GUI interface that
allowed one to type words into their form,
such as "(burger not cheeseburger) or (pizza
AND pepperoni)." They also allowed one to
form Boolean searches (except Hotbot as of
7/1/96, which promises to install this feature
later), i.e. they allowed the user to specify
combinations of words. In Alta Vista and
Lycos, one does this by adding a "+" or a "-"
sign before each word, or in Alta Vista you
can choose to use the very strict syntax
Boolean "advanced search." This advanced
search was by far the hardest to use, but also
the one most completely in the user's control
(except for OpenText). In most other
engines, you just use the words AND, NOT,
and OR to get Boolean logic.
By far the best service for carefully
specifying a search was Open Text. This
form has great menus, making a complex
Boolean search fast and easy. Best of all,
this service permits you to specify that you
want to search only titles or URLs. But then
there's Alta Vista's little known "keyword"
search syntax, now as powerful as Open
Text, but not as easy to use. You can
constrain a search to phrases in anchors,
pages from a specific host, image titles,
links, text, document titles, or URLs using
this feature with the syntax keyword: search-
word. There is an additional set of keywords
just for searching Usenet.
Which is the Best Search
Engine?
To decide which search engine I would
choose as the best, I decided that nothing but
useful results would count. Previous articles
have emphasized quantified measures for
speed and database sizes, but I found these
had little relevance for the best performance
in actual searches. By now, all engines have
great hardware and fast net links, and none
show any significant delay time to work on
your search or return the results. Instead, I
just came up with a few topics that
represented, I felt, tough but typical
problems encountered by people who work
on the net: First, I tried a search with
"background noise", a topic where a lot of
closely related but unwanted information
exists. Next, I tried a search for something
very obscure. Finally, I tried a search for
keywords which overlapped with a very,
very popular search keyword.
Example - Search Terms Which
Yield Too Many Matches
For the first type of search, I wanted to find
a copy of Wusage to download, free
software that lets you keep track of how
often your server or a specific page is
accessed, a common tool for HTML
developers. This site is hard to find because
output files are produced by the program on
every machine running it that have the string
"wusage" in their title and text. When I
simply typed "wusage" into search page
forms, Infoseek and Lycos were the only
engines to find the free version of the
software I wanted. (Note I gave no credit for
finding the version for sale. A careful search
of the sale version's page, did not produce
any links to the free version's download
site.) Infoseek's summaries were very poor,
however, and all matches had to be checked.
Always Search As
Specifically As
Possible
Most engines failed to find their quarry
because the search was too broad. After all,
how is the engine supposed to know I want
the free version? After spending a long time
to find out the exact name of what I wanted,
"wusage 3.2", Infoseek, Excite, Magellan,
and Lycos all found the site I was interested
in. Alta Vista, Hotbot, and OpenText
yielded nothing of interest on their first
page. Magellan came out the clear winner on
this search, as the site summary was by far
the best. Infoseek and Excite performed
well, but Lycos listed a much older version
of wusage first.
Think About Search Terms
It eventually occurred to me to search for
"wusage AND free" to find the free copy of
wusage. In some sense, Lycos was the
winner this time because the free version
was the first match listed; however, its
summary was not very useful. While it did a
better job than Infoseek, it didn't tell me
whether each site was relevant or not.
Magellan's response was very good, as it
included a link leading to the software on
the first page of matches, again with an
excellent summary. Yahoo and Alta Vista
also found it, but all these engines rated the
fee version higher than the free version.
OpenText did very well here, but only in
advanced search mode where it was possible
to specify that wusage must be in the title,
and "free" could be anywhere in the text.
Wusage3.2 was listed as the second of only
two entries - no digging here! Excite failed
to find the site at all, and HotBot found only
10 matches for statistics of a server in
Omaha.
Curiously, a search for "download wusage"
did not improve the results over the single-
word searches for any of the search engines!
(It may be time for rudimentary
standardized categories to be used on the
Web: e.g. this is a download archive, this is
an information only site, this is an
authoritative site, etc.) The lesson here may
just be "if at first you don't succeed..."
Results and
Performance
The most important measure of a search
engine is the quality of its search results.
While a complete user evaluation is beyond
the scope of this paper, our own experience
with Google has shown it to produce better
results than the major commercial search
engines for most searches. As an example
which illustrates the use of PageRank,
anchor text, and proximity, Figure 4 shows
Google's results for a search on "bill
clinton". These results demonstrates some of
Google's features. The results are clustered
by server. This helps considerably when
sifting through result sets. A number of
results are from the whitehouse.gov domain
which is what one may reasonably expect
from such a search. Currently, most major
commercial search engines do not return any
results from whitehouse.gov, much less the
right ones. Notice that there is no title for the
first result. This is because it was not
crawled. Instead, Google relied on anchor
text to determine this was a good answer to
the query. Similarly, the fifth result is an
email address which, of course, is not
crawlable. It is also a result of anchor text.
All of the results are reasonably high quality
pages and, at last check, none were broken
links. This is largely because they all have
high PageRank. The PageRanks are the
percentages in red along with bar graphs.
Finally, there are no results about a Bill
other than Clinton or about a Clinton other
than Bill. This is because we place heavy
importance on the proximity of word
occurrences. Of course a true test of the
quality of a search engine would involve an
extensive user study or results analysis
which we do not have room for here.
Instead, we invite the reader to try Google
for themselves at http://google.stanford.edu.
the total size of the repository is about 53
GB, just over one third of the total data it
stores. At current disk prices this makes the
repository a relatively cheap source of useful
data. More importantly, the total of all the
data used by the search engine requires a
comparable amount of storage, about 55 GB.
Furthermore, most queries can be answered
using just the short inverted index. With
Web Page Statistics
Number of Web
24 million
Pages Fetched

Storage Requirements 76.5

Number of Urls Seen
Aside from search quality, Google is million
designed to scale cost effectively to the size
of the Web as it grows. One aspect of this is Number of Email 1.7
to use storage efficiently. Table 1 has a Addresses million
breakdown of some statistics and storage
1.6
requirements of Google. Due to compression Number of 404's
million

Storage Statistics Table 1. Statistics

Total Size of Fetched better encoding and compression of the

147.8 GB
Pages Document Index, a high quality web search
engine may fit onto a 7GB drive of a new
Compressed Repository 53.5 GB PC.
Short Inverted Index 4.1 GB

Full Inverted Index 37.2 GB

Lexicon 293 MB
System Performance
It is important for a search engine to crawl
Temporary Anchor Data
6.6 GB and index efficiently. This way information
(not in total)
can be kept up to date and major changes to
Document Index Incl. the system can be tested relatively quickly.
9.7 GB
Variable Width Data For Google, the major operations are
Crawling, Indexing, and Sorting. It is
Links Database 3.9 GB
difficult to measure how long crawling took
Total Without overall because disks filled up, name servers
55.2 GB
Repository

108.7
Total With Repository
GB
crashed, or any number of other problems <html
xmlns="http://www.w3.org/1999/xhtml
which stopped the system. In total it took " xml:lang="en" lang="en" dir="ltr">
roughly 9 days to download the 26 million <head>
<meta http-equiv="Content-Type"
pages (including errors). However, once the
content="text/html; charset=utf-
system was running smoothly, it ran much 8" />
faster, downloading the last 11 million pages <title>search engine</title>
<meta http-equiv="pragma"
in just 63 hours, averaging just over 4 content="no-cache" />
million pages per day or 48.5 pages per <meta name="title"
second. We ran the indexer and the crawler content="www.cybwll.ch
searchengine" />
simultaneously. The indexer ran just faster <meta name="description"
than the crawlers. This is largely because we content="www.cybwell.ch" />
<meta name="robots"
spent just enough time optimizing the content="index, follow" />
indexer so that it would not be a bottleneck. <meta name="revisit-after"
These optimizations included bulk updates content="3 days" />
<meta name="author"
to the document index and placement of content="www.guide-bleu.ch" />
critical data structures on the local disk. The <meta name="publisher"
content="CYBWELL MEDIA GmbH,
indexer runs at roughly 54 pages per second. Steinhausen, Switzerland" />
The sorters can be run completely in <meta name="copyright"
parallel; using four machines, the whole content="www.cybwell.ch" />
<meta name="keywords"
process of sorting takes about 24 hours. content="" />
<link
Search Performance href="/cms/_styles/layout.css"
rel="stylesheet" type="text/css"
Improving the performance of search was media="all" />
not the major focus of our research up to this <link
href="/cms/_styles/general.css"
point. The current version of Google rel="stylesheet" type="text/css"
answers most queries in between 1 and 10 media="all" />
seconds. This time is mostly dominated by <link
href="/globalfiles/css/styles.css"
disk IO over NFS (since disks are spread rel="stylesheet" type="text/css"
over a number of machines). Furthermore, media="all" />
</head>
Google does not have any optimizations <body bgcolor="#FFFFFF">
such as query caching, subindices on
common terms, and other common <p>&nbsp;</p>
optimizations. We intend to speed up <table border="0" cellspacing="0"
Google considerably through distribution width="970" cellpadding="0">
<tr>
and hardware, software, and algorithmic
<td align="center" colspan="7"
improvements. Our target is to be able to width="970"><img border="0"
handle several hundred queries per second. src="/images/cybwell.gif">
<p>&nbsp;</p>
</td>
CODE FOR SEARCH ENGINE </tr>
<tr>
<td colspan="7" width="970"
align="center">
 <FORM method="POST" quality including page rank, anchor text, and
action="default.asp">
proximity information. Furthermore, Google
<INPUT class="forms" is a complete architecture for gathering web
name="q" size="22" pages, indexing them, and performing
id="layout1">&nbsp; <INPUT
class="forms" type="submit" search queries over them.
value="search" name="search"><br>
</FORM> Future Work
</td>
</tr> A large-scale web search engine is a
<tr> complex system and much remains to be
<td width="8" valign="top" done. Our immediate goals are to improve
align="left" bgcolor="#ECECEA"
bordercolor="#ECECEA"> search efficiency and to scale to
&nbsp;</td> approximately 100 million web pages. Some
<td width="718" valign="top"
align="left" bgcolor="#ECECEA" simple improvements to efficiency include
bordercolor="#ECECEA"> query caching, smart disk allocation, and
subindices. Another area which requires
</td> much research is updates. We must have
<td width="8" valign="top" smart algorithms to decide what old web
align="left" bgcolor="#ECECEA"
bordercolor="#ECECEA"> pages should be recrawled and what new
&nbsp; ones should be crawled. Work toward this
</td> goal has been done in One promising area of
<td width="18" valign="top"
align="left"></td> research is using proxy caches to build
<td width="8" valign="top" search databases, since they are demand
align="left"
bgcolor="#ECECEA">&nbsp;</td> driven. We are planning to add simple
<td width="222" valign="top" features supported by commercial search
align="left" bgcolor="#ECECEA" engines like boolean operators, negation,
bordercolor="#ECECEA">
<HR><H3>Related Searches</H3> and stemming. However, other features are
just starting to be explored such as relevance
<td width="8" valign="top"
align="left" bgcolor="#ECECEA"
feedback and clustering (Google currently
bordercolor="#ECECEA"> supports a simple hostname based
&nbsp; clustering). We also plan to support user
</tr>
</table> context (like the user's location), and result
summarization. We are also working to
</body>
extend the use of link structure and link text.
</html> Simple experiments indicate PageRank can
be personalized by increasing the weight of
Conclusions a user's home page or bookmarks. As for
Google is designed to be a scalable search link text, we are experimenting with using
engine. The primary goal is to provide high text surrounding links in addition to the link
quality search results over a rapidly growing text itself. A Web search engine is a very
World Wide Web. Google employs a rich environment for research ideas. We
number of techniques to improve search have far too many to list here so we do not
expect this Future Work section to become
much shorter in the near future.
High Quality Search
The biggest problem facing users of web
search engines today is the quality of the
results they get back. While the results are
often amusing and expand users' horizons,
they are often frustrating and consume
precious time. For example, the top result
for a search for "Bill Clinton" on one of the
most popular commercial search engines
was the Bill Clinton Joke of the Day: April
14, 1997. Google is designed to provide
higher quality search so as the Web
continues to grow rapidly, information can
be found easily. In order to accomplish this
Google makes heavy use of hypertextual
information consisting of link structure and
link (anchor) text. Google also uses
proximity and font information. While
evaluation of a search engine is difficult, we
have subjectively found that Google returns
higher quality search results than current
commercial search engines. The analysis of
link structure via PageRank allows Google
to evaluate the quality of web pages. The
use of link text as a description of what the
link points to helps the search engine return
relevant (and to some degree high quality)
results. Finally, the use of proximity
information helps increase relevance a great
deal for many queries.
Scalable Architecture
Aside from the quality of search, Google is
designed to scale. It must be efficient in both
space and time, and constant factors are very
important when dealing with the entire Web.
In implementing Google, we have seen
bottlenecks in CPU, memory access,
memory capacity, disk seeks, disk
throughput, disk capacity, and network IO.
Google has evolved to overcome a number
of these bottlenecks during various
operations. Google's major data structures
make efficient use of available storage
space. Furthermore, the crawling, indexing,
and sorting operations are efficient enough
to be able to build an index of a substantial
portion of the web -- 24 million pages, in
less than one week. We expect to be able to
build an index of 100 million pages in less
than a month.
A Research Tool
In addition to being a high quality search
engine, Google is a research tool. The
data Google has collected has already
resulted in many other papers
submitted to conferences and many
more on the way. Recent research
such as [Abiteboul 97] has shown a
number of limitations to queries about
the Web that may be answered
without having the Web available
locally. This means that Google (or a
similar system) is not only a valuable
research tool but a necessary one for a
wide range of applications. We hope
Google will be a resource for
searchers and researchers all around
the world and will spark the next
generation of search engine
technology.
REFERENCES
• Google Search Engine
http://google.stanford.edu/
• Search Engine Watch
http://www.searchenginewatch.
com/
• www.monash.com/spidap3.html
• http://www.webreference.com
• uk.searchengine.com/
• en.wikipedia.org/wiki/Web_sear
ch_engine
• searchengine.com/
• www.searchenginecommando.
com/articles/titles/16.html
• www.searchengineguide.com