Data Extraction from the Web and Security Issues

By

Siddu P. Algur
Head, Dept. of Information Science & Engineering S D M College of Engg. & Tech., Dharwad. siddu_p_algur@hotmail.com

CONENT
Motivation Solution Existing Approaches New Approach (VSAP Algorithm) Empirical Evaluation Experimental Results Conclusion

Motivation
Huge amount of information on the Internet. Data is distributed over Internet Presence of undesired data along with relevant information Requirement of data from various sources in local repository for further analysis

The Solution Web Mining

Definition - Data mining on web pages Data mining Extraction of useful information by observing patterns in the data Web mining can be used to extract the useful information from web pages. We use Web-Page Structure Mining

Web Mining : Data Mining On the Web

A Term coined by Etzioni in 1996

WEB MINING TAXONOMY Web Mining Web Usage Mining Web Structure Mining Web Content Mining

Web Structure Mining: The structure of a typical web graph consists of Web pages nodes and hyperlinks as edges connecting between two related pages. It can be regarded as the process of discovering structure information from the web Web Usage Mining: It focuses on techniques that could predict user behavior while the user interacts with the web. Web Content Mining: It emphasizes on the content of the web page. It is an automatic process that extracts pattern from web pages and goes beyond only the keyword extraction.

Web-Page Structure Mining

Defn : Identifying relevant information by observing the visual structure of a web page. From the visual structure of web pages, we can determine the position of relevant data on the web pages.

But
Retrieving relevant information from the web seems to be like Finding the Needle in the Haystack...

 The Web is highly volatile, distributed and heterogeneous. The Web is a huge chaotic information space without central authority.

The Web is noisy.

Existing Approaches & their Limitations

MDR Algorithm
( Mining Data Records from Web Pages )

DEPTA Algorithm
( Data Extraction using Partial

Tree Alignment ) VIPS Algorithm

( VIsion based Page Segmentation )

MDR Algorithm

Observations Data regions.

A group of data records are presented in a particular region of a page and formatted using similar HTML tags.

Data records.
A group of similar data records being placed in a specific region are under the same parent in a tag tree.

MDR Algorithm Steps:

Building a HTML Tag Tree of a
Page.

Mine Data Regions in page based upon Tag Tree & string comparison
Identify Data records from data regions.

Data Records

Data Region

TAG _TREE

4 GeneralizedNodes

Data Records

Limitations Of MDR Algorithm

Tag Dependent ( <TABLE>,<TR>,<TD>etc ) Extracts irrelevant data regions also along with the Relevant data region. Needs to do content mining to identify relevant data region. Highly prone to HTML tag-structure irregularities. Hence, fails in case of misuse of tags. Incorporates considerable time in building tag tree, traversing whole tag tree and string comparison.

DEPTA Algorithm Steps :

Given a page, it first segments the page using visual information, to identify each data record.

A novel partial tree alignment method is used to align and to extract corresponding data items from the discovered data records and put the data items in a database table.

Limitations Of DEPTA Algorithm

Constructing a tag tree using visual information has the limitation that , the tag tree can be built correctly only as long as the browser is able to render the page correctly. Tag-dependent and hence prone to tag-structure irregularities.

The computation time for constructing the tag tree and tree matching is an overhead.
Fails to identify the data records, in cases where there may be only a single record on page.

VIPS Algorithm
VIPS algorithm parses the HTML page and visual separators are detected in the parse tree.

The separators receive weights which are adjusted depending on constraints based on separator.

Finally, the content structure of the page is created, by merging visual blocks that are not divided by separators.

Limitations Of VIPS Algorithm

VIPS also does not correctly identify the data regions.

VIPS is dependent on number of heuristic rules which do not hold good for most of the pages.

Tool Bar Search and Filtering Panel

Data Region
Data Object 1 ( Data Record 1 )

Content Links

Data Object 2 ( Data Record 2 )

Data Object 3 ( Data Record 3 )

Advertise - ment Links

Data Object 4 ( Data Record 4 )

Copyright Statement

Layout of a typical Web Page

A Data Region containing 4 Data Records

Visual Structure

based

Analysis of web Pages

( The Proposed Approach )

Internet
HTML Page

Identifying the Data Region Largest Rectangle Identifier Co-ordinates of Bounding Rectangles Of All Tags

Parsing & Rendering Engine MSHTML.dll

Container Identifier

Data Region Identifier (Filter)

Relevant Data Region

VSAP

VSAP Algorithm

Steps :
Determine the co-ordinates of all the bounding rectangles. Identify the Data Region. Identify the Largest Rectangle. Identify the Container within the Largest Rectangle. Identify the Data Region containing the Data records within that Container.

Sample Web Page Of A Product related Web-site

HTML Parsing & Rendering Engine

HTML Page

Parsing & Rendering Engine

Co-ordinates of Bounding Rectangles Of All Tags

Component of every Browser Function Parse & Render HTML Pages Used to obtain bounding rectangles for each Tag.

E.g. MSHTML for Internet Explorer.

Web Page

Bounding Rectangles

Data Region Extractor

Co-ordinates of Bounding Rectangles Of All Tags

Data Region Extractor

Data Region

Made up of two components :

Container Identifier : Identifies the innermost tag which contains the data region Filter : Filters the identified container to get the data region

Container Identifier
Obtains largest bounding rectangle
Child of the BODY tag

Get smallest rectangle with area greater than half the area of largest bounding rectangle.

Web Page

Container Identified

Filter
Find Average Height of the children of the container Eliminate children whose height is less than average height

Container

Data Region

EMPIRICAL EVALUATION

Data Region Identification MDR Dependent on specific tags for identifying data regions. VSAP Identifies data regions independent of specific tags . Data Record Extraction MDR Identifying data records based on keyword search ( e.g . $ ) VSAP Identifying data records based on visual structure of the web page. Overall Time Complexity MDR O ( NK ) , N is total no. of nodes in tag tree and K is max. no. of tag nodes of a generalized node. DEPTA O ( k2 ) , k is the number of trees. VSAP O ( n ) , n is the no. of tag - comparisons made.

Performance Measures
Recall = Ec Nt Ec Nt Et
is the total number of records on the page . is the total number of records extracted.

Precision = Ec Et

is the total number of correctly extracted records.

Recall : The percentage of relevant data records identified

from the web page. Precision : The correctness of the data records identified.

EXPERIMENTAL RESULTS
URL Cor.
1. 2. 3. 4. 5. 6. 7. 8. 9. http://www.tigerdirect.com/. http://www.amazon.com/. http://www.cooking.com/. http://www.ebay.com/.. http://www.powells.com/.. http://www.barnesandnoble.com/. http://www.pricegrabber.com/.. http://www.shoebuy.com/. http://www.smartbuy.com/..

MDR
Wr. 36/ 0 17/25 13/3 30/0 47/1 30/0 0/25 12/84 15/10 38/1 12/0 6/15 Cor. 8 25 20 25 10 10 25 96 10 25 10 15 8 0 17 25 9 10 0 12 0 24 10 0

VSAP
Wr. 0/0 1/0 0/0 0/0 5/0 0/0 0/0 0/0 0/0 3/0 0/0 0/0

10. http://www.reviews.cnet.com/.. 11. http://www.nothingbutsoftware.com/ 12. http://www.refurbdepot.com/.

13. http://www.drugstore.com/.
14. http://www.bookpool.com/. 15. http://www.target.com/

15
10 0 140

14/0
7/0 0/12 277 / 176 33.5% 44.3%

15
10 12 316

0/0
0/0 1/0 10 / 0 96.93% 100%

Total
Recall Precision

Performance Comparison of MDR and VSAP

100% 96.93%

100 90 80 70 60 50 40 33.5% 30 20 10 0 Recall 44.3%

MDR VSAP

Precision

Data Record Extractor

DATA REGION

DATA RECORD EXTRACTOR

DATA RECORD

Extraction of data records is based on visual clues. Height of each record is obtained. Average height is calculated Data records whose height is greater than the average height is extracted.

DATA REGION

DATA RECORDS

Data Record Identifier

DATA RECORD DATA RECORD IDENTIFIER FLAT DATA RECORD

NESTED DATA RECORD

The flat record gives description of a single entity whereas the nested data record gives multiple description of a single entity

Identification of data records is essential in order to simplify the task of extracting the data items, which is very much needed for various applications. The Data Identifier determines the number of data fields in each data record within the data region. The data fields in flat records are less as compared to that of nested records. The number of fields in the nested data records is approximately 40% more than that of the flat records.

In fig1 the number of fields is 12 and in fig2 the number of fields is 7.The number of fields in fig1 is 58.3% more than the number of fields in fig2. Fig1 is a nested record and fig2 is flat record.

Extraction of Data Fields

Extraction of Data Fields is based on bounding rectangles. Each field is associated with a bounding rectangle. Data fields are extracted row by row The data fields are extracted and stored in a file.

Transferring the Data Items/Fields into the Database

Application
VSAP can be used by any application that requires the most relevant information of a web page VSAP can provide a platform for an application that requires to analyze related data from different sources on the web. VSAP can serve as an efficient replacement of MDR, which has already found its place in the industry.

Conclusion
Results show that Performance of VSAP is better than other existing algorithms VSAP is a novel & efficient method of web mining

References
[1] Baeza Yates, R. Algorithms for string matching: A survey. ACM SIGIR Forum,

23(3-4):3458, 1989.
[2] J. Hammer, H. Garcia Molina, J. Cho, and A. Crespo . Extracting semistructured information from the web. In Proc. of the Workshop on the

Management of Semi-structured Data, 1997.

[3] D. Embley, Y. Jiang, and Y. K . Ng. Record-boundary discovery in Web documents. ACM SIGMOD Conference, 1999 [4] Kushmerick, N. Wrapper Induction: Efficiency and Expressiveness. Artificial Intelligence, 118:15-68, 2000. Clustering-based Approach to Integrating Source Query ] [5] Chang, C-H., Lui, S-L. IEPAD: Information Extraction Based on Pattern Discovery. WWW-01, 2001. ] [6] Crescenzi, V., Mecca, G. and Merialdo, P. ROADRUNNER: Towards Automatic Data Extraction from Large Web Sites. VLDB-01, 2001.] [7] Y.Yang, H. Zhang. HTML Page Analysis based on Visual Cues. 6th

International Conference on Document Analysis and Recognition, 2001.

[8] D. Buttler, L. Liu, C. Pu. A Fully Automated Object Extraction System for the World Wide Web. International Conference on Distributed Computing Systems

(ICDCS 2001), 2001

[9] Bing Liu , Kevin chen-chuan chang, Editorial: Special issue on web content mining, WWW 02, 2002. [10] Liu, B., Grossman, R. and Zhai, Y. Mining Data Records in Web Pages. KDD03, 2003.

[11] Cai, D., Yu, S., Wen, J.-R. and Ma, W.-Y. (2003). Extracting Content Structure for Web Pages based on
Visual Representation, Asia Pacific Web Conference (APWeb 2003), pp. 406417. [12] A. Arasu, H. Garcia-Molina, Extracting structured data from web pages, ACM

SIGMOD 2003, 2003

[13] J. Wang, F. H Lochovsky. Data Extraction and Label Assignment for Web Databases.WWW conference, 2003. [14] H. Zhao, W. Meng, Z. Wu, Raghavan, Clement Yu. Fully Automatic Wrapper Generation For Search Engines, International WWW conference 2005, May 10-

14,2005, Japan. ACM 1-59593-046-9/05/005

[15] Zhai, Y., Liu, B. Web Data Extraction Based on Partial Tree Alignment ,

WWW-05, 2005, May 10-14, 2005, Chiba, Japan. ACM 1-59593-046-9/05/0005

[16] Kosala R., Hendrick Blockeel. Web Mining Research : A Survey , SIGKDD

Explorations, ACM SIGKDD , July 2000.

Snapshot
Amazon.com

Result

By VSAP

By MDR

Cooking.com

Result

By VSAP

By MDR

Tigerdirect.com

Result

By VSAP

By MDR

Overall VSAP Algorithm

Algorithm VSAP ( HTML document )
Begin
Set maxRect = NULL Set dataRegion = NULL FindMaxRect (BODY); FindDataRegion ( maxRect ); FilterDataRegion ( dataRegion ); End

Algorithm to identify Largest Rectangle Procedure FindMaxRect ( BODY )

Begin for each child of BODY tag begin find the co-ordinates of the bounding rectangle for the child if the area of the bounding rectangle > area of maxRect then maxRect = child endif end End

Algorithm to identify Largest Rectangle

Procedure FindMaxRect ( BODY )
Begin for each child of BODY tag begin find the co-ordinates of the bounding rectangle for the child if the area of the bounding rectangle > area of maxRect then maxRect = child endif end End

Algorithm to identify Container of Data Region Procedure FindDataRegion ( maxRect )

Begin
ListChildren = depth first listing of the children of the tag associated with maxRect for each tag in ListChildren begin If area of bounding rectangle of tag > half the area of maxRect then If area of bounding rectangle dataRegion > area of bounding rectangle of tag then dataRegion = tag endif endif end

End

Algorithm to Filter Data region from the container

Algorithm FilterDataRegion (dataRegion )
Begin for each child of dataRegion begin totalHeight += height of the bounding rectangle of child end avgHeight = totalHeight / no of children of dataRegion for each child of dataRegion begin If height of childs bounding rectangle < avgHeight then Remove child from dataRegion endif end End

Thank You