Finding Hidden Components

'
NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis
NSF-CBMS Lecture Series

Lecture 1:
Finding Hidden Components
By Data Analysis
Agenda
$ '
Hidden Components
Principal Components
Beyond Principal Components
David Donoho
Statistics Department
Stanford University
Statistical Analysis of Images
% &
Look Ahead
&
From Newtons Notebook
$ '
(b) Prism
% &
'
(a) Newton
I.A Hidden Components in Light
&
'
I.B Hidden Components in Heat
Figure 2: Fourier
$ '
Diffusion of Heat
f (, t), for [0, 2)
f (, t = 0), for [0, 2)
Initial Temperature Profile:
Profile at later time:
Shows
less variation
more smoothness
eventual uniformity
Stages of Diffusion
Fouriers Insight
cn exp{in}
n (t) cn exp{in}
Expand in Sinusoidal Components

X
f (, t = 0) =
Profile made of waves.
f (, t) =
Profile at later time:
Damping factor:
n (t) = exp{n2 t}
Decreases rapidly with wavelength, time
% &
&
% &
$ '
'
1.5
0.5
-0.5
&
f(,t)
'
1.5
1
0.5
0
-0.5
0
Initial Density t=0
Components t=0
10
Components t=.0469
f(,t=.)
10
Component
Diffusion
Fourier1
Fourier
1.5
(cn (t))
(cn (0))
Modern Viewpoint I
f (, 0)
f (, t)
0.5
-0.5
&
'
Component
&
f(,0)
Density at t=.0469
n (t)
$ '
Interpretation
Fourier Analysis is a kind of Prism
Hidden Components: Sinusoidal Components
Post Facto: Sinusoids Fundamental
Dominant trend, 19th century Mathematical Physics
10
Tn,m cm
12
% &
11
$ '
nm
Modern Viewpoint II
Linear Transformation
(T c)n =
Diagonal Transformation
(T c)n = Tn,n cn
Diagonal no interaction between components.
% &
'
f (, 0)
Fourier1
Fourier
(cn (t))
(cn (0))
Modern Viewpoint III
Diffusion
f (, t)
Diagonal
13
$ '
II. Principal Components
14
(b) Lo`eve
% &
(a) Karhunen
&
hn , m i = n,m
(t, s)n (s)ds = n n (t)
1 2 3 . . .
16
Eigenvectors
Under general conditions, orthobasis 1 , 2 , . . .,
Cov(X(t), X(s)) = E(X(t) EX(t))(X(s) EX(s))
(t, s) its covariance:
X(t), t T a random process in L2 (T )
Karhunen-Lo`
eve Transform
$ '
(b) Hotelling
Eigenvalues
% &
15
'
(a) Pearson
&
'
f
KLT1
KLT
(dn )
(cn )
Modern Viewpoint
Covariance
f
17
$ '
KLT Representation
X(t), t T a random process in L2 (T )
Xn n (t)
Cov(Xn , Xm ) = n n,m
V arXn = n
Xn = hX, n i, its n-th coordinate in KL basis

Variances
Covariances
Representation
X(t) =
18
% &
here the Xn are orthogonal.

&
20
$ '
Noise Removal
Y (t) observed data
X(t) desired signal to recover
Y (t) = X(t) + N (t)
wn Yn n (t)
n
.
n + n
N (t) Gaussian noise process, with

Same KL basis as X: (n )
wn =
E{X(t)|Y } =
Eigenvalues (n )
where
% &
19
'
Some Key Applications

Noise Removal Wiener
Data Compression Shannon
&
'
KLT
Modern Viewpoint
Y
KLT1
n)
(X
(Yn )
Expectation
E{X(t)|Y }
wn
21
$ '
Data Compression
22
0.1
0.2
0.3
0.4
Brownian Bridge
0.5
t
0.6
0.7
24
% &
30
25
20
15
10
-5
-10
-15
-20
% &
23
&
0.9
$ '
0.8
'
Key Example
B(t) Brownian Bridge.

Gaussian
B(0) = B(1) = 0
(t, s) = min(t, s) ts.
Closely related to Brownian Motion
&
B(t)
'
2 sin(nt)
X
Zn n (t)
KLT of Brownian
n (t) =
n (0) = n (1) = 0
n = Const/n2
Representation:
B(t) =
Principal Scores Zn indep N (0, n ).
25
$ '
III. Beyond Principal Components

Independent Components
26
Sparse Components
% &
Sinusoids are the Principal Components of Brownian Bridge.

&
Independent Components of Gaussian Process

For Gaussian Process, IC = PC.
28
n n (t)
$ '
X
n
Proof:
If (Xn ) uncorrelated & jointly Gaussian, the (Xn ) are

independent.
For a Gaussian Process, the principal scores are uncorrelated

and jointly Gaussian.
% &
27
'
X(t) =
III.A Independent Components
n Independent
n Basis Functions
Remarks
Need not exist
Need not be orthogonal
No procedure in general
&
'
t
t1
t
t<
Ramp: A NonGaussian Process

Scalar RV U (0, 1).
X(t, ) =
Stochastic Process (Y. Meyer)
Properties:
EX(t) = 0
29
0.1
0.2
0.3
0.4
Ramp
0.5
t
0.6
0.7
$ '
1
0.8
0.6
0.4
0.2
-0.2
0.8
0.9
30
-0.4
% &
X(1) = X(0) = 0
&
0.1
0.1
0.2
0.2
0.3
0.3
0.4
0.4
Brownian Bridge
0.5
t
Ramp
0.5
t
0.6
0.7
0.7
0.8
0.8
0.9
0.9
32
$
50
40
30
20
10
-10
-20
1
0.8
0.6
0.4
0.2
-0.2
-0.4
0.6
% &
31
$ '
&
Second-Order Equivalent to Brownian Bridge!
(t, s) = Cov(X(t), X(s)) = min(t, s) ts
Covariance of Ramp
'
Ramp(t)
B(t)
Ramp(t)
'
Contrast
Principal Components are second-order independent, but
perhaps not independent.
33
$ '
III.B Approximate ICA
Attempt to approximate Higher-Order Independence

Form Cumulant Tensor of order > 2
34
Attempt to Diagonalize Tensor
% &
Independent Components are independent, but perhaps cannot

be found or do not exist.
&
E(X1 X2 ) E(X3 X4 )
E(X3 ) E(X1 X2 X4 ) E(X4 ) E(X1 X2 X3 )
E(X1 ) E(X2 X3 X4 ) E(X2 ) E(X1 X3 X4 )
E(X1 X2 X3 X4 )
Independence and Cumulant Tensors
Let Xn = hX, n i coefficient in basis (n ).

Cumulant Tensor K4, 4-way array:
K4, (i1 , i2 , i3 , i4 ) = Cum4 (Xi1 , Xi2 , Xi3 , Xi4 )
If (X ) are independent random variables, K4, is hyperdiagonal:

n
0 not all indices same

? i 1 = i2 = i 3 = i4
K4, (i1 , i2 , i3 , i4 ) =
36
E(X1 X3 ) E(X2 X4 ) + E(X1 X4 ) E(X2 X3 )
$ '
2(EX1 )(EX2 )(EX3 )(EX4 )
% &
35
'
Fourth-order Joint Cumulants
Kurtosis: K4 (X) = E(X )4 3V ar(X)2
Cum4 (X1 , . . . , X4 )
&
'
Numerical Experiment
Finite Ramp Process:
T = {1/32, . . . , 32/32}
= {1/32, . . . , 32/32}
Database X(i, j) =
X(i , tj ).
N = 32 realizations X(i, )
N = 32 variables X(, t)
Use Jean-Francois Cardoso JADE Algorithm
Attempt Diagonalization of Empirical Cumulant Tensor
Moves 90% of Kurtic Energy on Diagonal.
37
$ '
20
20
20
20
20
Results from JADE analysis of Ramp
-1
10
20
-1
10
20
-1
10
-1
10
-1
10
-1
10
-1
10
20
20
20
-1
10
-1
20
20
20
20
20
20
20
0
-1
10
0
-1
10
0
-1
10
0
-1
10
0
-1
10
0
-1
10
0
20
0
-1
-1
10
20
20
20
20
20
20
20
-1
10
0
-1
10
0
-1
10
0
-1
10
0
-1
10
0
20
0
-1
-1
10
20
20
20
20
20
20
0
-1
10
0
-1
10
0
-1
10
0
-1
10
0
-1
10
0
-1
10
-1
10
0
-1
Jadelet 16
Jadelet 9
Comparisons
Jadelet 5
0.5
0.5
0.5
-1
0.5
20
-1
20
-0.5
-1
Wavelet 23
-0.5
20
0.5
Wavelet 21
-0.5
-1
Wavelet 12
0.5
0.5
-1
0.5
20
-1
20
-0.5
-1
-0.5
20
-0.5
-1
-0.5
-0.5
Wavelet 25
20
38
40
%
39
% &
20
20
20
20
20
20
20
20
Jadelet 13
&
20
0
-1
10
0
-1
$ '
20
% &
'
20
-1
10
20
0
-1
10
20
0
-1
10
0
20
0
-1
10
0
-1
10
20
20
20
0
-1
10
0
-1
10
0
-1
10
20
-1
10
-1
10
-1
10
-1
10
20
20
20
20
-1
10
-1
10
-1
10
-1
10
20
20
20
20
-1
10
-1
10
-1
10
-1
10
20
20
20
20
-1
10
-1
0
20
-1
10
0
-1
-1
10
20
0
0
-1
10
-1
-1
10
-1
10
-1
10
20
Daubechies Nearly Symmetric Wavelets
&
'
Interpretations
Hidden Components discovered by JADE
Dyadic Scales
Near-Translates
Approx. Daubechies Nearly Symmetric
41
$ '
Xn n (t)
III.D Sparse Components Analysis

X(t) =
(Xn ) a sparse vector: few large entries.
n a basis or dictionary to be found
Goal choose (n ) to optimize sparsity

Remarks:
Need not exist
No general definition
42
Quasi-Wavelets are the almost-independent components of Ramp
% &
No general algorithm
&
cnj nj k2
44
$ '
j=1
m
X
Comparison of SCA with PCA (cont)

SCA (One approach)
B(m) (X, (n )) best m-term approximation,
kX B(m) (X, (n ))k2 = argmin(nj ) kX
argmin(n ) EkX B(m) (X, (n ))k22
Find Basis optimizing
% &
43
'
Comparison of SCA with PCA
PCA
m
X
n=1
cn n
A(m) (X, (n )) first m-term approximation,

A(m) X =
argmin(n ) EkX A(m) (X, (n ))k22
Find Basis (n ) optimizing
&
'
X
j,k j,k (t)
Example of Approximate SCA: Ramp

Wavelet Representation
X(t) =
n
(j,k ) Orthonormal Wavelets basis, 3 vanishing Moments.

(j,k ) sparse vector: few large entries.
At level j at most C nonzeros, k
With 54 coefficients can represent to accuracy 106
160
45
$ '
Sparsity Comparison
Wavelet Basis (quasi-SCA)
With 54 coefficients can represent to accuracy 106
kX B(m) (X, wavelets)k2 C1 exp(C2 m)

Fourier Basis (PCA)
Need 1902 coefficients for accuracy 106 .
kX A(m) (X, sinusoids)k2 C/ m
kX B(m) (X, sinusoids)k2 C/ m
1
0.8
0.6
0.4
0.2
-0.2
-0.4
-2
-4
-6
-8
-10
0.2
0.6
Ramp
0.4
0.4
0.6
0.8
0.8
Wavelet Transform of Ramp
0.2
Coefficient Amplitude
0.5
-0.5
10
-2
10
-4
10
-6
10
200
100
200
k-th largest coefficient
Wavelet Transform
Sine Transform
Ordered Transform Coefficients
100
Sine Transform of Ramp
46
48
140
47
% &
300
&
120
Compression Numbers for DST and Wavelet Analysis of Ramp
80
Sine Transform
60
Wavelet Transform
40
300
200
$ '
20
180
'
-1
-2
-3
-7
-6
-5
-4
-8
100
n
% &
Ramp(t)
&
Log(c )
'
Second-Order
Independence
High-Order
ICA
First-m Term
Approximation
Best-m Term
SCA
Contrast of ICA/SCA/PCA
New Concept
PCA Concept
49
Aspect
Immense
ICA
Approximation Theory
Rarified
SCA
$ '
Popularity
Statistical
Comparison of ICA/SCA
Mode
Plausibility
Many Heuristics
None
Few Heuristics
Sufficient
Harmonic Analysis
Algorithms
50
Approximation
% &
Independence
&
|K4, (i1 , i2 , i3 , i4 )|p )1/p
52
i1 ,i2 ,i3
Almost-Diagonality of Cumulant Tensors
$ '
51
'
Similar Outcome from ICA & SCA: Ramp
i1
Interesting case; p 0; says each slice sparse.
kK4, kAD(p) = sup(
Wavelets give very high rate of m-term Approximation.
Theorem. For circular Ramp process on [0, 2), Daubechies nearly

symmetric wavelets with 3 vanishing moments
% &
Almost-diagonal in sense of good decay away from hyper-diagonal.
kK4, kAD(p) <
Wavelets almost diagonalize the cumulant tensor.

Can find qualitatively same outcome by either principle.
&
'
Interpretation
Study 1 d objects with singularities
They are nongaussian
PCA would give sinsuoidal components
ICA & SCA give wavelet components
53
$ '
Retina
...
V1
IV. Visual Neuroscience
Information Flow in the Visual System

Image
V1 Component of Visual Cortex:
Linear Receptive Fields Hubel and Wiesel.
Neural Outputn = hImage, n i

Properties of (n )
Localized
Oriented
...
54
What if we study 2 d objects Images?
% &
Wavelet/Gabor Models
&
(b) Field
56
(a) Barlow
IV.A Two Slogans
$ '
% &
55
'
Caricature of V1
&
'
Barlow
Visual System does a remarkable job of image compression.
107 bits/sec at input
20 40 bits/sec at end of processing
57
$ '
Field
Structure of natural images drives structure of visual system
Redundancy of natural images could teach visual system how

to be sparse
58
Statistical analysis of images and neurodevelopment? (Cats)
% &
Visual system takes highly redundant data, removes the

redundancy, produces nearly independent data.
&
60
Depict results
Sparse Components Analysis
Database of patches
Extract 16 16 patches.
Collection of natural images
Appeared in Nature, 1996
Olshausen-Field Experiment
$ '
(b) Field
% &
59
'
(a) Olshausen
Sparse Components of Natural Images
&
'
Database of Image Patches

Collection of natural images
Extract 16 16 patches.
Patch P (p) = (P (p) (s, t), 1 s, t 16
X(p, ) = entries of P (p) in row-major order.
Result: X an N by 256 matrix
61
$ '
j (p)j
Empirical Sparse Components Analysis

Want X(p, j)
kX(p, )
Xj (p)j k2 + S((p))
Find both coefficients j (p) and basis j .

Want j (p) sparse.
Formal Objective
X
(j ) (j (p))
min min
Sparsity measure
log(1 + j2 )
62
% &
S() =
&
64
Interpretation
$ '
Multi-Oriented
Multiscale
Bandpass
Initially, wavelets. Later not in title.
Clearly (to me) not wavelets.
% &
63
'
&
66
65
$ '
'
% &
Beamlets
Curvelets
Ridgelets
Similarities to Recent CHA ideas
Not Gabor too many scales
Not Wavelets too many directions
What Are These Things?
&
68
van der Schaaf and van Hateren
Hyvarinen
Lewicki-Sejnowksi
Hundreds of papers
NIPS Conferences
ICA-2000 Conference
Independent Components Analysis
$ '
(b) Sejnowksi
% &
67
'
(a) Bell
Independent Components of Natural Images
&
'
Independent Components of Natural Movies
69
$ '
A Role for Math?
PCA Data Analysis finds Sinusoids
ICA/SCA Data Analysis of 1-d Discontinuities finds Wavelets
ICA/SCA Data Analysis of 2-d Discontinuities finds ???
ICA/SCA Data Analysis of 3-d Discontinuities finds ???
70
Little Prospect of substantially larger data analysis
% &
Figure 8: Hans van Hateren
&
Wed.
Curvelets
Thurs.
Combined
Beamlets
Fri.
72
$ '
Tues.
Ridgelets
Curvelets
Lecture Plan
Mon.
Ridgelets
Curvelets
71
'
Lectures to Come
Analogy between Statistical Analysis of Image Databases&
Harmonic Analysis of Image Models.
Data Anal.
Ridgelets
Common Factor: Sparse Representation

Point Singularities vs. Line Singularities
Math Anal.
% &
Line Singularities vs. Curve Singularities

Harmonic Analysis of Line & Curve Singularities
What Lies Beyond
&
'
Whats The Connection?
73
Whats at Stake
$ '
Applications
Data Compression
Noise Removal
Detection, Classification
New Approximation Theory
New Connections with Mathematics
New Research in Harmonic Analysis
74
(b) Imagelets
% &
(a) Ridgelets
&
(b) X. Huo
76
(a) E. Cand`es
Collaborators (2)
$ '
% &
75
'
Figure 9: Iain Johnstone
Collaborators (1)
&
'
Collaborators (3ab)
(b) Coifman
77
$ '
(b)
Wald
en
Collaborators (3cd)
(a) Israeli
78
(a) Averbuch
% &
(b) Y. Hel-Or
80
&
(a) H. Hel-Or
Collaborators (5)
Collaborators (4)
(b) E. Arias
% &
79
$ '
(c) O. Levi
'
(a) G. Flesia
&
'
(a) Stodden
Thanks ...
(b) Duncan
&
81

Finding Hidden Components

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Finding Hidden Components

Uploaded by

Copyright:

Available Formats

'

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Lecture Series

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

Statistical Analysis of Images

From Newtons Notebook

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

I.A Hidden Components in Light

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

I.B Hidden Components in Heat

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

f (, t), for [0, 2)

f (, t = 0), for [0, 2)

Initial Temperature Profile:

Profile at later time:

Expand in Sinusoidal Components

Profile made of waves.

Profile at later time:

Decreases rapidly with wavelength, time

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

Initial Density t=0

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

Fourier Analysis is a kind of Prism

Hidden Components: Sinusoidal Components

Post Facto: Sinusoids Fundamental

Dominant trend, 19th century Mathematical Physics

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

Diagonal no interaction between components.

Modern Viewpoint III

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

II. Principal Components

(t, s)n (s)ds = n n (t)

Under general conditions, orthobasis 1 , 2 , . . .,

Cov(X(t), X(s)) = E(X(t) EX(t))(X(s) EX(s))

(t, s) its covariance:

X(t), t T a random process in L2 (T )

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

X(t), t T a random process in L2 (T )

Xn = hX, n i, its n-th coordinate in KL basis

here the Xn are orthogonal.

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

Y (t) = X(t) + N (t)

N (t) Gaussian noise process, with

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

Some Key Applications

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

B(t) Brownian Bridge.

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis

III. Beyond Principal Components

Sinusoids are the Principal Components of Brownian Bridge.

Independent Components of Gaussian Process

NSF-CBMS Talk 1: Finding Hidden Components by Data Analysis