Web Harvesting

RAHUL.MADANU 09BK1A0535 CSE-A

Web Harvest

Introduction A Web crawler is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing.

A web search engine is software code that is designed to search for information on the World Wide Web.
A database is an organized collection of data.

Web Harvest

Existing System

Web Harvest

Q: How does a search engine know that all these pages contain the query terms? A: Because all of those pages have been crawled

Web Harvest

Motivation for crawlers

Support universal search engines (Google, Yahoo, MSN/Windows Live, Ask, etc.) Vertical (specialized) search engines, e.g. news, shopping, papers, recipes, reviews, etc. Business intelligence: keep track of potential competitors, partners Monitor Web sites of interest Evil: harvest emails for spamming, phishing Can you think of some others?

Web Harvest

Many names
Crawler Spider Robot (or bot) Web agent Wanderer, worm, And famous instances: googlebot, scooter, slurp, msnbot,

Web Harvest

A crawler within a search engine

Web googlebot Page repository

Text & link analysis

Query

hits

Text index
Ranker

PageRank

Web Harvest

Page Rank

Web Harvest

Page Rank Probability -

Web Harvest

Proposed System

Web Harvest

10

Aim :
Have to set higher memory range. Eliminate all file not found. Removing negative dictionary.

Need to obtain Base URL.

Multi-processing or multi-threading.

Web Harvest

11

Recovering Issues Dont want to fetch same page twice or Save up in Marked list. Soft fail for timeout, server not responding, file not found, and other errors. Noise words that do not carry meaning should be eliminated (stopped) before they are indexed E.g. in English: AND, THE, A, AT, OR, ON, FOR, etc Need to obtain Base URL from HTTP header

Base: http://www.cnn.com/linkto/ Relative URL: intl.html Absolute URL:

http://www.cnn.com/linkto/intl.html

Overlap the above delays by fetching many pages concurrently Can use multi-processing or multi-threading

Web Harvest

12

Policy Coverage Freshness Trade-off (Subjective)

Web Harvest

13

Algorithm Algorithm for classifying a crawler data into the database

Bayesian approaches are a fundamentally important DM technique. Given the probability distribution, Bayes classifier can provably achieve the optimal result. Bayes Classifier is that it assumes all attributes are independent of each other.

Web Harvest

14

Basic crawlers
This is a sequential crawler Seeds can be any list of starting URLs Order of page visits is determined by frontier data structure Stop criterion can be anything

Graph traversal (BFS or DFS?)

Breadth First Search

Implemented with QUEUE (FIFO)

Finds pages along shortest paths

If we start with good pages, this keeps us close; maybe other good stuff

Depth First Search

Implemented with STACK (LIFO)

Wander away (lost in cyberspace)

Web Harvest

16

Breadth-first crawlers
BF crawler tends to crawl high-PageRank pages very early Therefore, BF crawler is a good baseline to gauge other crawlers
Average Number of Pages Crawled

Crawler ethics and conflicts

Crawlers can cause trouble, even unwillingly, if not properly designed to be polite and ethical lexical analysis is the process of converting a sequence of characters into a sequence of tokens. http://foo.com/woo/foo/woo/foo/woo

For example, sending too many requests in rapid succession to a single server can amount to a Denial of Service (DoS) attack!

Server administrator and users will be upset Crawler developer/admin IP address may be blacklisted

Web Harvest

18

Crawler etiquette (important!)

Spread the load, do not overwhelm a server

Make sure that no more than some max. number of requests to any single server per unit time, say < 1/second

Honor the Robot Exclusion Protocol

A server can specify which parts of its document tree any crawler is or is not allowed to crawl by a file named robots.txt placed in the HTTP root directory, e.g. http://www.indiana.edu/robots.txt Crawler should always check, parse, and obey this file before sending any requests to a server More info at:

http://www.google.com/robots.txt

http://www.robotstxt.org/wc/exclusion.html

Web Harvest

19

A Basic crawler in Java

import java.io.BufferedReader; import java.io.InputStreamReader; import java.net.URL; public class Main { public static void main(String[] args) { try { URL my_url = new URL("http://www.blogspot.com/"); BufferedReader br = new BufferedReader(new InputStreamReader(my_url.openStream())); String strTemp = ""; while(null != (strTemp = br.readLine())){ System.out.println(strTemp); } } catch (Exception ex) { ex.printStackTrace(); } } }

Web Harvest

20

Xml (Used) Extensible Markup Language (XML) is a markup language that defines a set of rules for encoding documents in a format that is both humanreadable and machine-readable. Many application programming interfaces (APIs) have been developed to aid software developers with processing XML data, and several schema systems exist to aid in the definition of XMLbased languages. Any forms of Database.
Web Harvest 21

Comparison Fields Time out N Dictionary Dynamic Pages URL Table Search Google Accepted Accepted Doubled Relative Big Table Page Rank Web Harvest Eliminated Eliminated Updated Base Bayes Table Limit Rank

Web Harvest

22

Web Harvest

23

Conclusion Web Harvesting Engine Marketing has one of the lowest costs per customer acquisition. Web Harvesting Engine is one of the most cost efficient ways to reach a target market for a small, medium or large business. Traditional marketing such as catalog mail, trade magazines, direct mail, TV or radio involves passive participation by your audience and targeting can very greatly from one medium to another.
Web Harvest 24

Queries ?
Web Harvest 25