DATAWAREHOUSING AND MINING

BY G.RAJESH CHANDRA Department of ECM K l University

EVOLUTION OF DATABASE TECHNOLOGY

1960s (Primitive File Processing)

Data collection, database creation, IMS and network DBMS

1970s to early 1980s (DBMS)

Relational data model, relational DBMS implementation ,SQL, OLTP,User Interfaces.etc

RDBMS, advanced data models (extended-relational, OO, deductive, etc.) Application-oriented DBMS (spatial, scientific, engineering, etc.) Data mining, data warehousing, multimedia databases, and Web databases Stream data management and mining Data mining and its applications

1980s: to Present (Advanced Data Bases)

1990s: (Advanced Data Analysis)

2000s

WHY MINE DATA? COMMERCIAL VIEWPOINT

Lots of data is being collected and warehoused

Web data, e-commerce purchases at department/ grocery stores Bank/Credit Card transactions Provide better, customized services for an edge (e.g. in Customer Relationship Management)

Competitive Pressure is Strong

WHAT IS DATA MINING..?

Data mining (sometimes called data Discovery or Knowledge Discovery Data) is the process of analyzing data from different perspectives and summarizing it into useful information. Extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) patterns or knowledge from huge amount of data

WHY MINE DATA? SCIENTIFIC VIEWPOINT

Data collected and stored at enormous speeds (GB/hour)

remote sensors on a satellite telescopes scanning the skies microarrays generating gene expression data scientific simulations generating terabytes of data

Traditional techniques infeasible for raw data Data mining may help scientists

in classifying and segmenting data in Hypothesis Formation

EXAMPLES: WHAT IS (NOT) DATA MINING?

What is not Data What is Data Mining?

Mining?

Look up phone
number in phone directory

Certain names are more

prevalent in certain US locations (OBrien, ORurke, OReilly in Boston area) Group together similar documents returned by search engine according to their context (e.g. Amazon rainforest, Amazon.com,)

Query a Web
search engine for information about Amazon

DATA MINING IS ALSO CALLED AS..?

Knowledge discovery (mining) in databases (KDD), knowledge extraction, data/pattern analysis, data archeology, data dredging, information harvesting, business intelligence, etc. Real Time Example Gold Mining

DATA WARE HOUSE = COLLECTION OF DATA BASES

WE HAVE TO USE DIFFERENT METHODS

RAW DATA =DATA BASES + NOISE DATA

DATA SELECTION AND TRANSFORMATION

DATA CLEANING AND INTEGRATION

DATA MINING

PATTERN EVALUATION

KNOWLEDGE REPRASENTATION

KNOWLEDGE REPRASENTATION

December 26, 2013

KNOWLEDGE DISCOVERY (KDD) PROCESS

Data

miningcore of knowledge discovery process

Task-relevant Data Data Warehouse Selection

Pattern Evaluation

Data Mining

Data Cleaning
Data Integration Databases