You are on page 1of 82

Data Mining: Data

Lecture Notes for Session 2

Introduction to Data Mining


by
Tan, Steinbach, Kumar
By Dr. K. Satyanarayan Reddy

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

What is Data?

Collection of data objects and


their attributes.

An attribute is a property or
characteristic of an object

Attributes

Examples: eye color of a


person, temperature, etc.

Attribute is also known as


variable, field, characteristic, or
feature & dimension.
A collection of
describe an Object

Attributes

Objects

Object is also known as


Record, Point, Case, Vector,
Pattern,
Sample,
Entity,
Observation or Instance.
Tan,Steinbach, Kumar

Introduction to Data Mining

Tid Refund Marital


Status

Taxable
Income Cheat

Yes

Single

125K

No

No

Married

100K

No

No

Single

70K

No

Yes

Married

120K

No

No

Divorced 95K

Yes

No

Married

No

Yes

Divorced 220K

No

No

Single

85K

Yes

No

Married

75K

No

10

No

Single

90K

Yes

60K

10

4/18/2004

Data
Definition 1: An attribute is a property or
characteristic of an object that may vary; either
from one object to another or from one time to
another.
Definition 2: A measurement scale is a rule
(function) that associates a numerical or symbolic
value with an attribute of an object.
The process of Measurement is the application of a
measurement scale to associate a value with a
particular attribute of a specific object.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

Attribute Values

Attribute values are numbers or symbols assigned


to an attribute.

Distinction between attributes and attribute values


Same attribute can be mapped to different attribute
values

Example: height can be measured in feet or meters

Different attributes can be mapped to the same set of


values
Example: Attribute values for ID and age are integers.
But properties of attribute values can be different

ID has no limit but age has a maximum and minimum


value.
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

Measurement of Length

The way you measure an attribute is somewhat may not match the
attributes properties.
5

B
7

2
C

D
10

15

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

Types of Attributes

There are different types of attributes


Nominal

Examples: ID numbers, eye color, zip codes

Ordinal

Examples: rankings (e.g., taste of potato chips on a scale


from 1-10), grades, height in {tall, medium, short}

Interval

Examples: calendar dates, temperatures in Celsius or


Fahrenheit.

Ratio

Examples: temperature in Kelvin, length, time, counts

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

Properties of Attribute Values

The type of an attribute depends on which of the


following properties it possesses:

Distinctness:
Order:
Addition:
Multiplication:

Nominal attribute: distinctness


Ordinal attribute: distinctness & order
Interval attribute: distinctness, order & addition
Ratio attribute: all 4 properties

Tan,Steinbach, Kumar

=
< >
+ */

Introduction to Data Mining

4/18/2004

Types of Attributes
Nominal and ordinal attributes are collectively
referred to as Categorical or Qualitative attributes.
As the name suggests, qualitative attributes, such as
employee ID, lack most of the properties of
numbers.
The remaining two types of attributes, Interval and
Ratio, are collectively referred to as Quantitative or
Numeric Attributes.
Quantitative attributes are represented by numbers
and have most of the properties of numbers.
Note: Quantitative attributes can be integer-valued or
continuous.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

Attribute
Type

Description

Examples

Nominal

The values of a nominal attribute are


just different names, i.e., nominal
attributes provide only enough
information to distinguish one object
from another. (=, )

zip codes, employee


ID numbers, eye color,
sex: {male, female}

mode, entropy,
contingency
correlation, 2 test

Ordinal

The values of an ordinal attribute


provide enough information to order
objects. (<, >)

hardness of minerals,
{good, better, best},
grades, street numbers

median, percentiles,
rank correlation,
run tests, sign tests

Interval

For interval attributes, the


differences between values are
meaningful, i.e., a unit of
measurement exists.
(+, - )

calendar dates,
temperature in Celsius
or Fahrenheit

mean, standard
deviation, Pearson's
correlation, t and F
tests

For ratio variables, both differences


and ratios are meaningful. (*, /)

temperature in Kelvin,
monetary quantities,
counts, age, mass,
length, electrical
current

geometric mean,
harmonic mean,
percent variation

Ratio

Operations

Attribute
Level

Transformation

Comments

Nominal

Any permutation of values

If all employee ID numbers


were reassigned, would it
make any difference?

Ordinal

An order preserving change of


values, i.e.,
new_value = f(old_value)
where f is a monotonic function.

Interval

new_value =a * old_value + b
where a and b are constants

An attribute encompassing
the notion of good, better
best can be represented
equally well by the values
{1, 2, 3} or by { 0.5, 1,
10}.
Thus, the Fahrenheit and
Celsius temperature scales
differ in terms of where
their zero value is and the
size of a unit (degree).

Ratio

new_value = a * old_value

Length can be measured in


meters or feet.

Discrete and Continuous Attributes

Discrete Attribute
Has only a finite or countably infinite set of values
Examples: zip codes, counts, or the set of words in a collection of
documents
Often represented as integer variables.
Note: binary attributes are a special case of discrete attributes

Continuous Attribute
Has real numbers as attribute values
Examples: temperature, height, or weight.
Practically, real values can only be measured and represented using a finite
number of digits.
Continuous attributes are typically represented as floating-point variables.

Asymmetric Attributes: Binary attributes where only non-zero values are important
are called asymmetric binary attributes.

Example: If the number of credits associated with each course is recorded, then the
resulting data set will consist of asymmetric discrete or continuous attributes.
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

12

Types of Data Sets

Record
Data Matrix
Document Data
Transaction Data

Graph

World Wide Web


Molecular Structures

Ordered
Spatial Data
Temporal Data
Sequential Data
Genetic Sequence Data

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

13

Important Characteristics of Structured Data

Dimensionality: The dimensionality of a data set is the number of attributes that the objects
in the data set possess.

Curse of Dimensionality: Data with a small number of dimensions tends to be qualitatively


different than moderate or high-dimensional data.
The difficulties associated with analyzing high-dimensional data are sometimes referred to as the
curse of dimensionality.

Sparsity

Only Presence Counts: For some data sets, such as those with asymmetric features, most
attributes of an object have values of 0; in many cases fewer than 1% of the entries are nonzero.
In practical terms, sparsity is an advantage because usually only the non-zero values need to be
stored and manipulated. This results in significant savings with respect to computation time and
storage.

Resolution

Patterns depend on the scale

It is frequently possible to obtain data at different levels of resoIution, and often the properties of
the data are different at different resolutions.

Example: The surface of the Earth seems very uneven at a resolution of a few meters, but is
relatively smooth at a resolution of tens of kilometers.
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

14

Record Data

Data that consists of a collection of records(data


objects), each of which consists of a fixed set of
data fields (attributes).
Tid Refund Marital
Status

Taxable
Income Cheat

Yes

Single

125K

No

No

Married

100K

No

No

Single

70K

No

Yes

Married

120K

No

No

Divorced 95K

Yes

No

Married

No

Yes

Divorced 220K

No

No

Single

85K

Yes

No

Married

75K

No

10

No

Single

90K

Yes

60K

10

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

15

Transaction or Market Basket Data


Transaction Data is a special type of record data, where each record (transaction) involves
a set of items.
Consider a grocery store where a set of products purchased by a customer during one
shopping trip constitutes a Transaction, while the individual products that were
purchased are the Items.

This type of data is called Market Basket Data because the items in each record are the
products in a person's "market basket."

Transaction data is a collection of sets of items, but it can


be viewed as a set of records whose fields are asymmetric
attributes.
Often, the attributes are binary, indicating whether or not an
item was purchased, but the attributes can be discrete or
continuous, such as the number of items purchased or the
amount spent on those items.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

16

Data Matrix

If data objects have the same fixed set of numeric


attributes, then the data objects can be thought of as
points in a multi-dimensional space, where each
dimension represents a distinct attribute

Such data set can be represented by an m by n matrix,


where there are m rows, one for each object, and n
columns, one for each attribute
Projection
of x Load

Projection
of y load

Distance

Load

Thickness

10.23

5.27

15.22

2.7

1.2

12.65

6.25

16.22

2.2

1.1

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

17

The Sparse Data Matrix

A Sparse Data Matrix is a special case


of a Data Matrix in which the attributes
are of the same type and are
asymmetric; i.e., only non-zero values
are important.
Transaction data is an example of a
sparse data matrix that has only 0 - 1
entries.
A common example is Document Data.
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

18

Document Data

Each document becomes a `term' vector,


each term is a component (attribute) of the vector,
the value of each component is the number of times
the corresponding term occurs in the document.

team

coach

pla
y

ball

score

game

wi
n

lost

timeout

season

Document 1

Document 2

Document 3

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

19

Transaction Data

A special type of record data, where


each record (transaction) involves a set of items.
For example, consider a grocery store. The set of
products purchased by a customer during one
shopping trip constitute a transaction, while the
individual products that were purchased are the items.
TID

Items

Bread, Coke, Milk

2
3
4
5

Beer, Bread
Beer, Coke, Diaper, Milk
Beer, Bread, Diaper, Milk
Coke, Diaper, Milk

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

20

Graph Based Data


A graph can sometimes be a convenient and powerful representation for
data.
There are two specific cases:
(1)

The graph captures relationships among data objects and

(2)

The data objects themselves are represented as graphs.

Data with Relationships among Objects: The relationships among objects


frequently convey important information. In such cases, the data is often
represented as a graph.

The data objects are mapped to nodes of the graph, while the
relationships among objects are captured by the links between objects
and link properties, such as direction and weight.
Data with Objects which are Graphs: If objects have structure i.e. the
objects contain sub-objects that have relationships, then such objects are
frequently represented as graphs.
e.g.: The structure of chemical compounds can be represented by a
graph, where the nodes are atoms and the links between nodes are
chemical bonds.
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

21

Graph Data contd

Examples: Generic graph and HTML Links

<a href="papers/papers.html#bbbb">
Data Mining </a>
<li>
<a href="papers/papers.html#aaaa">
Graph Partitioning </a>
<li>
<a href="papers/papers.html#aaaa">
Parallel Solution of Sparse Linear System of Equations </a>
<li>
<a href="papers/papers.html#ffff">
N-Body Computation and Dense Linear System Solvers

2
1

5
2
5

Linked Web pages


Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

22

Chemical Data

Benzene Molecule: C6H6

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

23

Ordered Data

Sequences of Transactions: For some types of data, the


attributes have relationships that involve order in time or space.
Items/Events

An element of
the sequence
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

24

Ordered Data contd.

Sequential data, also referred to as temporal data, can be thought of


as an extension of record data, where each record has a time
associated with it.

Consider a retail transaction data set that also stores the time at which
the transaction took place.
This time information makes it possible to find patterns such as "candy
sales peak before Halloween." A time can also be associated with
each attribute.

Sequential Transaction Data


Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

25

Ordered Data contd.

Sequence Data Sequence data consists of a data set that is a


sequence of individual entities, such as a sequence of words or
letters. It is quite similar to sequential data, except that there are
no time stamps; instead, there are positions in an ordered
sequence.

Example: Genomic sequence data:


the genetic information of plants and
animals can be represented in the
form of sequences of nucleotides
that are known as genes.
Many of the problems associated
with genetic sequence data involve
predicting similarities in the structure
and function of genes from
similarities in nucleotide sequences.

Tan,Steinbach, Kumar

GGTTCCGCCTTCAGCCCCGCGCC
CGCAGGGCCCGCCCCGCGCCGTC
GAGAAGGGCCCGCCTGGCGGGCG
GGGGGAGGCGGGGCCGCCCGAGC
CCAACCGAGTCCGACCAGGTGCC
CCCTCTGCTCGGCCTAGACCTGA
GCTCATTAGGCGGCAGCGGACAG
GCCAAGTAGAACACGCGAAGCGC
TGGGCTGCCTGCTGCGACCAGGG

Introduction to Data Mining

4/18/2004

26

Ordered Data contd.

Time Series Data: Time series data is a special type of sequential data in which each
record is a time series, i.e., a series of measurements taken over time.
e.g.: A financial data set might contain objects that are time series of the daily prices of
various stocks.
e.g.: Consider Figure below, which shows a time series of the average monthly
temperature for Minneapolis during the years 1982 to 1994. When working with
temporal data, it is important to consider temporal autocorrelation; i.e., if two
measurements are close in time, then the values of those measurements are often
very similar.

Temperature time series


Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

27

Ordered Data
Spatial Data: Some objects have spatial attributes, such as positions or areas,
as well as other types of attributes.
e.g.: Weather data (precipitation, temperature, pressure) that is collected for a
variety of geographical locations.
An important aspect of spatial data is spatial autocorrelation; i.e., objects that
are physically close tend to be similar in other ways as well. Thus, two points
on the Earth that are close to each other usually have similar values for
temperature and rainfall.
Spatio-Temporal Data:

Average Monthly
Temperature
of
land and ocean

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

28

Data Quality
Data Mining focuses on the
Detection and correction of data quality problems and
Use of algorithms that can tolerate poor data quality.

The first step, detection and correction, is often called Data


Cleaning.
What kinds of data quality problems?
How can we detect problems with the data?
What can we do about these problems?
Examples of data quality problems:
Noise and outliers

missing values
duplicate data
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

29

Issues with Data


Measurement and Data Collection Issues: Perfect data cant be expected. There may be
problems due to human error, limitations of measuring devices, or flaws in the data
collection process.
Values or even entire data objects may be missing or there may be spurious or duplicate
objects; i.e., multiple data objects that all correspond to a single "real" object.
e.g.: There might be two different records for a person who has recently lived at two
different addresses.

Even if all the data is present and "looks fine," there may be inconsistencies-a person
has a height of 2 meters, but weighs only 2 kilograms.
Measurement and Data Collection Errors: The term Measurement error refers to any
problem resulting from the measurement process it may be the value recorded differs
from the true value.
For continuous attributes, the numerical difference of the measured and true value is
called the error.
The term data collection error refers to errors such as omitting data objects or attribute
values, or inappropriately including a data object.
e.g.: A study of animals of a certain species might include animals of a related species
that are similar in appearance to the species of interest. Both measurement errors and
data collection errors can be either systematic or random.
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

30

Noise

Noise refers to modification of original values


Examples: distortion of a persons voice when talking
on a poor phone and snow on television screen

Two Sine Waves


Tan,Steinbach, Kumar

Introduction to Data Mining

Two Sine Waves + Noise


4/18/2004

31

Precision, Bias and Accuracy


Precision: The closeness of repeated measurements (of the
same quantity) to one another.
Bias: A systematic quantity being measured.
Precision is often measured by the Standard Deviation of a set
of values, while Bias is measured by taking the difference
between the Mean of the set of values and the known value
of the quantity being measured.
Accuracy: The closeness of measurements to the true value of
the quantity being measured.
Accuracy depends on precision and bias, but there is no
specific formula for accuracy in terms of these two
quantities.
Only Accuracy is counted in terms of the of Significant Digits.
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

32

Outliers

Outliers are data objects with characteristics that


are considerably different than most of the other
data objects in the data set

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

33

Missing Values

Reasons for Missing Values


Information
is
not
(e.g., people decline to give their age and weight)
Attributes
may
not
be
applicable
to
(e.g., annual income is not applicable to children)

all

cases

Handling Missing Values

collected

Eliminate Data Objects


Estimate Missing Values
Ignore the Missing Value During Analysis
Replace with all possible values (weighted by their probabilities)

Inconsistent Values: Data can contain inconsistent values.

Consider an address field, where both a zip code and city are
listed, but the specified zip code area is not contained in that city.
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

34

Duplicate Data

Data set may include data objects that are


duplicates, or almost duplicates of one another
Major issue when merging data from heterogeous
sources

Examples:
Same person with multiple email addresses

Data cleaning
Process of dealing with duplicate data issues

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

35

Issues Related to Applications


There are a few general issues:
Timeliness: Some data starts to age as soon as it has been
collected.
If the data is out of date, then so are the models and
patterns that are based on it.
Relevance: The available data must contain the information
necessary for the application.
Knowledge about the Data: The data sets are accompanied
by documentation that describes different aspects of the
data; the quality of this documentation can either aid or
hinder the subsequent analysis.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

36

Data Preprocessing
Aggregation
Sampling
Dimensionality Reduction
Feature subset selection
Feature creation
Discretization and Binarization
Attribute Transformation

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

37

Aggregation

Combining two or more attributes (or objects) into


a single attribute (or object)

Purpose
Data reduction

Reduce the number of attributes or objects

Change of scale

Cities aggregated into regions, states, countries, etc

More stable data

Aggregated data tends to have less variability

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

38

Variation of Precipitation in Australia

Aggregation

Standard Deviation of Average


Monthly Precipitation
Tan,Steinbach, Kumar

Introduction to Data Mining

Standard Deviation of Average


Yearly Precipitation
4/18/2004

39

Sampling

Sampling is the main technique employed for data


selection.
It is often used for both the preliminary investigation of the
data and the final data analysis.

Statisticians sample because obtaining the entire set of


data of interest is too expensive or time consuming.

Sampling is used in data mining because processing the


entire set of data of interest is too expensive or time
consuming.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

40

Sampling

The key principle for effective sampling is the


following:
using a sample will work almost as well as using the
entire data sets, if the sample is representative

A sample is representative if it has approximately the


same property (of interest) as the original set of data

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

41

Types of Sampling

Simple Random Sampling


There is an equal probability of selecting any particular item

Sampling without replacement


As each item is selected, it is removed from the population

Sampling with replacement


Objects are not removed from the population as they are
selected for the sample.
In sampling with replacement, the same object can be picked up
more than once

Stratified sampling
Split the data into several partitions; then draw random samples
from each partition

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

42

Sample Size

8000 points

Tan,Steinbach, Kumar

2000 Points

Introduction to Data Mining

500 Points

4/18/2004

43

Sample Size

What sample size is necessary to get at least one


object from each of 10 groups.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

44

Curse of Dimensionality

When
dimensionality
increases, data becomes
increasingly sparse in the
space that it occupies

Definitions of density and


distance between points,
which
is
critical
for
clustering
and
outlier
detection, become less
meaningful

Randomly generate 500 points


Compute difference between max and min
distance between any pair of points

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

45

Dimensionality Reduction

Purpose:
Avoid curse of dimensionality
Reduce amount of time and memory required by data
mining algorithms
Allow data to be more easily visualized
May help to eliminate irrelevant features or reduce
noise

Techniques
Principle Component Analysis
Singular Value Decomposition
Others: supervised and non-linear techniques

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

46

Dimensionality Reduction: PCA

Goal is to find a projection that captures the


largest amount of variation in data
x2

x1
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

47

Dimensionality Reduction: PCA


Find the eigenvectors of the covariance matrix
The eigenvectors define the new space

x2
e

x1
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

48

Dimensionality Reduction: ISOMAP


By: Tenenbaum, de Silva,
Langford (2000)

Construct a neighbourhood graph


For each pair of points in the graph, compute the shortest
path distances geodesic distances

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

49

Dimensionality Reduction: PCA


= 206
The
curse Dimensions
of Dimensions
Dimensions
==160
120
10
40
80
dimensionality
refers
to
the
phenomenon that
many types of data
analysis become
significantly harder
as
the
dimensionality of
the data increases.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

50

Linear Algebra Techniques for Dimensionality Reduction

Principal Components Analysis (PCA) is a linear


algebra technique for continuous attributes that
finds new attributes (principal components) that
are:1. linear combinations of the original attributes,
2. orthogonal (perpendicular) to each other, and
3. capture the maximum amount of variation in the
data.
Singular Value Decomposition (SVD) is a linear
algebra technique that is related to PCA and is
also commonly used for dimensionality reduction.
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

51

Feature Subset Selection

Another way to reduce dimensionality of data

Redundant features
duplicate much or all of the information contained in
one or more other attributes
Example: purchase price of a product and the amount
of sales tax paid

Irrelevant features
contain no information that is useful for the data
mining task at hand
Example: students' ID is often irrelevant to the task of
predicting students' GPA

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

52

Feature Subset Selection contd.


Techniques:

Brute-force approach:
Try

Embedded approaches:

Feature selection occurs naturally as part of the data mining algorithm

Filter approaches:

all possible feature subsets as input to data mining algorithm

Features are selected before data mining algorithm is run

Wrapper approaches:

Use the data mining algorithm as a black box to find best subset of attributes

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

53

An Architecture for Feature Subset Selection


As the number of subsets can be enormous and it is impractical to examine
them all, some sort of stopping criterion is necessary.
Once a subset of features has been selected, the results of the target data
mining algorithm on the selected subset should be validated.
Another approach for Validation is to use a number of different feature
selection algorithms to obtain subsets of features and then compare the
results of running the data mining algorithm on each subset.
Feature Weighting: It is an alternative to keeping or eliminating
features. More important features are assigned a higher weight, while
less important features are given a lower weight.
Feature Creation: It is frequently possible to create, from the original
attributes, a new set of attributes that captures the important information
in a data set much more effectively.
Feature Extraction: The creation of a new set of features from the
original raw data is known as feature extraction.
e.g.: Extracting facial features of human, from set of Photographs.
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

54

Feature Creation

Create new attributes that can capture the important


information in a data set much more efficiently than the original
attributes

Three general methodologies:


Feature Extraction
domain-specific

Mapping Data to New Space: A totally different view of the


data can reveal important and interesting features.
Feature Construction: Sometimes the features in the
original data sets have the necessary information, but it is
not in a form suitable for the data mining algorithm.

Tan,Steinbach, Kumar

combining features
Introduction to Data Mining

4/18/2004

55

Mapping Data to a New Space

Fourier Transform

Wavelet Transform

Two Sine Waves

Tan,Steinbach, Kumar

Two Sine Waves + Noise

Introduction to Data Mining

Frequency

4/18/2004

56

Discretization Using Class Labels


it is often necessary to transform a continuous attribute into a categorical
attribute (discretization), and both continuous and discrete attributes may
need to be transformed into one or more binary attributes (binarization).

Entropy based approach:

3 categories for both x and y

Tan,Steinbach, Kumar

Introduction to Data Mining

5 categories for both x and y

4/18/2004

57

Discretization Without Using Class Labels

Data

Equal interval width

Equal frequency
Tan,Steinbach, Kumar

K-means
Introduction to Data Mining

4/18/2004

58

Attribute Transformation

A function that maps the entire set of values of a


given attribute to a new set of replacement values
such that each old value can be identified with
one of the new values
Simple functions: xk, log(x), ex, |x|
Standardization and Normalization

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

59

Similarity and Dissimilarity

Similarity
Numerical measure of how alike two data objects are.
Is higher when objects are more alike.
Often falls in the range [0,1]

Dissimilarity
Numerical measure of how different are two data
objects
Lower when objects are more alike
Minimum dissimilarity is often 0
Upper limit varies

Proximity refers to a similarity or dissimilarity

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

60

Similarity and Dissimilarity contd.


p and q are the attribute values for two data objects.

Similarity/Dissimilarity for Simple Attributes

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

61

Euclidean Distance

Euclidean Distance

dist

( pk

k 1

qk )

Where n is the number of dimensions (attributes) and pk and qk


are, respectively, the kth attributes (components) or data
objects p and q.

Standardization is necessary, if scales differ.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

62

Euclidean Distance

point
p1
p2
p3
p4

p1

p3

p4

1
p2

0
0

y
2
0
1
1

p1
p1
p2
p3
p4

x
0
2
3
5

0
2.828
3.162
5.099

p2
2.828
0
1.414
3.162

p3
3.162
1.414
0
2

p4
5.099
3.162
2
0

Distance Matrix
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

63

Minkowski Distance

Minkowski Distance is a generalization of Euclidean


Distance

dist ( | pk
k 1

qk

1
r r
|)

Where r is a parameter, n is the number of dimensions


(attributes) and pk and qk are, respectively, the kth attributes
(components) or data objects p and q.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

64

Minkowski Distance: Examples

r = 1. City block (Manhattan, taxicab, L1 norm) distance.


A common example of this is the Hamming distance, which is just the
number of bits that are different between two binary vectors

r = 2. Euclidean distance

. supremum (Lmax norm, L norm) distance.


This is the maximum difference between any component of the vectors

Do not confuse r with n, i.e., all these distances are


defined for all numbers of dimensions.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

65

Minkowski Distance

point
p1
p2
p3
p4

x
0
2
3
5

y
2
0
1
1

L1
p1
p2
p3
p4

p1
0
4
4
6

p2
4
0
2
4

p3
4
2
0
2

p4
6
4
2
0

L2
p1
p2
p3
p4

p1

p2
2.828
0
1.414
3.162

p3
3.162
1.414
0
2

p4
5.099
3.162
2
0

L
p1
p2
p3
p4

p1

p2

p3

p4

0
2.828
3.162
5.099
0
2
3
5

2
0
1
3

3
1
0
2

5
3
2
0

Distance Matrix
Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

66

Mahalanobis Distance

mahalanobis( p, q)

( p q)

( p q)

is the covariance matrix of


the input data X

1
j ,k

n 1i

( X ij

X j )( X ik

For red points, the Euclidean distance is 14.7, Mahalanobis distance is 6.


Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

67

Xk)

Covariance Matrix:

0.3 0.2
0.2 0.3
C
B

Mahalanobis Distance

A: (0.5, 0.5)

B: (0, 1)
A

C: (1.5, 1.5)

Mahal(A,B) = 5
Mahal(A,C) = 4

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

68

Common Properties of a Distance

Distances, such as the Euclidean distance,


have some well known properties.
1.

d(p, q)
0
for all p and q and d(p, q) = 0 only if
p = q. (Positive definiteness)

2.

d(p, q) = d(q, p) for all p and q. (Symmetry)

3.

d(p, r)
d(p, q) + d(q, r)
(Triangle Inequality)

for all points p, q, and r.

where d(p, q) is the distance (dissimilarity) between


points (data objects), p and q.

A distance that satisfies these properties is a


metric

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

69

Common Properties of a Similarity

Similarities, also have some well known


properties.
1.

s(p, q) = 1 (or maximum similarity) only if p = q.

2.

s(p, q) = s(q, p)

for all p and q. (Symmetry)

where s(p, q) is the similarity between points (data


objects), p and q.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

70

Similarity Between Binary Vectors

Common situation is that objects, p and q, have only


binary attributes

Compute similarities using the following quantities


M01 = the number of attributes where p was 0 and q was 1
M10 = the number of attributes where p was 1 and q was 0
M00 = the number of attributes where p was 0 and q was 0
M11 = the number of attributes where p was 1 and q was 1

Simple Matching and Jaccard Coefficients


SMC = number of matches / number of attributes

= (M11 + M00) / (M01 + M10 + M11 + M00)

J = number of 11 matches / number of not-both-zero attributes values


= (M11) / (M01 + M10 + M11)

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

71

SMC versus Jaccard: Example


p= 1000000000
q= 0000001001
M01 = 2 (the number of attributes where p was 0 and q was 1)
M10 = 1 (the number of attributes where p was 1 and q was 0)

M00 = 7 (the number of attributes where p was 0 and q was 0)


M11 = 0 (the number of attributes where p was 1 and q was 1)

SMC = (M11 + M00)/(M01 + M10 + M11 + M00) = (0+7) / (2+1+0+7) = 0.7

J = (M11) / (M01 + M10 + M11) = 0 / (2 + 1 + 0) = 0

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

72

Cosine Similarity

If d1 and d2 are two document vectors, then


cos( d1, d2 ) = (d1
where

d2) / ||d1|| ||d2|| ,

indicates vector dot product and || d || is the length of vector d.

Example:
d1 = 3 2 0 5 0 0 0 2 0 0
d2 = 1 0 0 0 0 0 0 1 0 2
d1

d2= 3*1 + 2*0 + 0*0 + 5*0 + 0*0 + 0*0 + 0*0 + 2*1 + 0*0 + 0*2 = 5

||d1|| = (3*3+2*2+0*0+5*5+0*0+0*0+0*0+2*2+0*0+0*0)0.5 = (42) 0.5 = 6.481


||d2|| = (1*1+0*0+0*0+0*0+0*0+0*0+0*0+1*1+0*0+2*2) 0.5 = (6) 0.5 = 2.245

cos( d1, d2 ) = .3150


Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

73

Extended Jaccard Coefficient (Tanimoto)

Variation of Jaccard for continuous or count


attributes
Reduces to Jaccard for binary attributes

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

74

Correlation
Correlation measures the linear relationship
between objects
To compute correlation, we standardize data
objects, p and q, and then take their dot product

pk

( pk

mean( p)) / std ( p)

qk

( qk

mean( q)) / std ( q)

correlation( p, q)
Tan,Steinbach, Kumar

Introduction to Data Mining

p q
4/18/2004

75

Visually Evaluating Correlation

Scatter plots
showing the
similarity from
1 to 1.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

76

General Approach for Combining Similarities

Sometimes attributes are of many different


types, but an overall similarity is needed.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

77

Using Weights to Combine Similarities

May not want to treat all attributes the same.


Use weights wk which are between 0 and 1 and sum
to 1.

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

78

Density

Density-based clustering require a notion of


density

Examples:
Euclidean density

Euclidean density = number of points per unit volume

Probability density

Graph-based density

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

79

Euclidean Density Cell-based

Simplest approach is to divide region into a


number of rectangular cells of equal volume and
define density as # of points the cell contains

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

80

Euclidean Density Center-based

Euclidean density is the number of points within a


specified radius of the point

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

81

THANK YOU

Tan,Steinbach, Kumar

Introduction to Data Mining

4/18/2004

82

You might also like