Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Exploring Nonnegative Matrix Factorization

Holly Jin
LinkedIn Corp
and

Michael Saunders
Systems Optimization Laboratory, Stanford University

MMDS08
Workshop on Algorithms for Modern Massive Data Sets
Stanford University, June 2528, 2008

MMDS08

1/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Outline
1

Introduction

SNMF motivation

Sparse NMF

Basis Pursuit DeNoising (BPDN)

SNMF results

Application examples

MMDS08

2/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Introduction
Many applications (signal processing, page rank, imaging)
seek sparse solutions to square or rectangular systems
Ax b
x sparse

MMDS08

3/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Introduction
Many applications (signal processing, page rank, imaging)
seek sparse solutions to square or rectangular systems
Ax b
x sparse
Basis Pursuit Denoising (BPDN) seeks sparsity by solving

min

MMDS08

1
2 kAx

bk2 + kxk1

3/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Introduction
Many applications (signal processing, page rank, imaging)
seek sparse solutions to square or rectangular systems
Ax b
x sparse
Basis Pursuit Denoising (BPDN) seeks sparsity by solving

min

1
2 kAx

bk2 + kxk1

Sparse Nonnegative Matrix Factorization (SNMF) involves

square or rectangular systems

A WH
W,H 0

MMDS08

low rank and sparse

3/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Introduction
Many applications (signal processing, page rank, imaging)
seek sparse solutions to square or rectangular systems
Ax b
x sparse
Basis Pursuit Denoising (BPDN) seeks sparsity by solving

min

1
2 kAx

bk2 + kxk1

Sparse Nonnegative Matrix Factorization (SNMF) involves

square or rectangular systems

A WH
W,H 0

low rank and sparse

Perhaps BPDN can find very sparse approximate W , H

MMDS08

3/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Motivation for Sparse NMF Solver

MMDS08

4/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

An example

Grouping similar items in a grocery store

MMDS08

5/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

An example

Grouping similar items in a grocery store

Customer check-out receipts:
Flour Balloon Beer Sugar Chip
Customer1
Customer2
Customer3
Customer4
Customer5

MMDS08

0
0
5
0
10

3
2
0
20
0

8
5
1
40
1

0
1
10
2
10

1
0
0
1
1

5/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Extract features using SVD

0.1852 0.0225
0.1179
0.0282

0.0338
0.6126
U=

0.9744 0.0492
0.0356
0.7881

45.9457
0

0
17.7720

S =
0
0

0
0
0
0

0.0114
0.6158
0.4414 0.0560

V =
0.8949 0.0341
0.0601
0.7842
0.0260
0.0403
MMDS08

0.1457
0.1676
0.7649
0.0030
0.6046

0.9394
0.2529
0.0793
0.2098
0.0570

0
0
2.9418
0
0

0
0
0
1.1892
0

0.7550
0.0144
0.0164
0.6042
0.2540

0.1455
0.7337
0.3441
0.0478
0.5656

0.2480
0.9451

0.1794

0.0645
0.0945

0
0

0
0.2783

0.1719
0.5134

0.2818

0.1191
0.7831
6/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Truncated SVD
Choose k = 2 principal features according to matrix S:

0.1852
0.1179

Uk =
0.0338
0.9744
0.0356

45.9457
Sk =
0

0.0114
0.4414

Vk =
0.8949
0.0601
0.0260

MMDS08

0.0225
0.0282

0.6126

0.0492
0.7881

0
17.7720

0.6158
0.0560

0.0341

0.7842
0.0403

7/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Truncated SVD
Choose k = 2 principal features according to matrix S:

0.1852
0.1179

Uk =
0.0338
0.9744
0.0356

45.9457
Sk =
0

0.0114
0.4414

Vk =
0.8949
0.0601
0.0260

0.0225
0.0282

0.6126

0.0492
0.7881

0
17.7720

0.6158
0.0560

0.0341

0.7842
0.0403

Error kA USV T k = 1.5e-14

vs kA Uk Sk VkT k = 2.9 (minimized Frobenius norm)
MMDS08

7/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Clustering and ranking

Row and0column clustering
and
0 rankings:
1
1
Rr =

MMDS08

(1, 1)
B(2, 1)
B
B(4, 1)
B
@(3, 2)
(5, 2)

2
3C
C
1C
C
2A
1

Rc =

(2, 1)
B(3, 1)
B
B(1, 2)
B
@(4, 2)
(5, 2)

2
1C
C
2C
C
1A
3

8/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Clustering and ranking

Row and0column clustering
and
0 rankings:
1
1
Rr =

(1, 1)
B(2, 1)
B
B(4, 1)
B
@(3, 2)
(5, 2)

2
3C
C
1C
C
2A
1

Rc =

(2, 1)
B(3, 1)
B
B(1, 2)
B
@(4, 2)
(5, 2)

2
1C
C
2C
C
1A
3

Order of k preserves the ranking of cluster importance

MMDS08

8/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Clustering and ranking

Row and0column clustering
and
0 rankings:
1
1
Rr =

(1, 1)
B(2, 1)
B
B(4, 1)
B
@(3, 2)
(5, 2)

2
3C
C
1C
C
2A
1

Rc =

(2, 1)
B(3, 1)
B
B(1, 2)
B
@(4, 2)
(5, 2)

2
1C
C
2C
C
1A
3

Order of k preserves the ranking of cluster importance

Cluster example meanings:
Customer
Customer
Customer
Customer
Customer

MMDS08

1
2
3
4
5

Flour

Balloon

Beer

Sugar

Chip

0
0
5
0
10

3
2
0
20
0

8
5
1
40
1

0
1
10
2
10

1
0
0
1
1

8/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Clustering and ranking

Row and0column clustering
and
0 rankings:
1
1
Rr =

(1, 1)
B(2, 1)
B
B(4, 1)
B
@(3, 2)
(5, 2)

2
3C
C
1C
C
2A
1

Rc =

(2, 1)
B(3, 1)
B
B(1, 2)
B
@(4, 2)
(5, 2)

2
1C
C
2C
C
1A
3

Order of k preserves the ranking of cluster importance

Cluster example meanings:
Customer
Customer
Customer
Customer
Customer

1
2
3
4
5

Flour

Balloon

Beer

Sugar

Chip

0
0
5
0
10

3
2
0
20
0

8
5
1
40
1

0
1
10
2
10

1
0
0
1
1

Features extraction:
Customers
Products
MMDS08

Partying
1, 2, 4
Balloon, Beer

Baking
3, 5
Flour, Sugar, Chip
8/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

NMF
A = WH,

MMDS08

A W k Hk

9/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

NMF
A = WH, A Wk Hk
Factorization not unique:

(via Chih-Jen Lins NMF solver)

1
0
1.2850
B 0.4711 0.8065C
B
C
C
Wk = B
B 8.4380 0.0365C
@ 0.0217 6.7563A
10.8476
0

0.7968
0
0.0928
Hk =
0
2.9321 5.9337
0

MMDS08

1.0214
0.2885

0.0567
0.1667

9/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

NMF
A = WH, A Wk Hk
Factorization not unique:

(via Chih-Jen Lins NMF solver)

1
0
1.2850
B 0.4711 0.8065C
B
C
C
Wk = B
B 8.4380 0.0365C
@ 0.0217 6.7563A
10.8476
0

0.7968
0
0.0928
Hk =
0
2.9321 5.9337
0

1.0214
0.2885

0.0567
0.1667

Clustering and Ranking:

0

Rr =

MMDS08

(1, 2)
B(2, 2)
B
B(4, 2)
B
@(3, 1)
(5, 1)

1
2
3C
C
1C
C
2A
1

Rc =

(2, 2)
B(3, 2)
B
B(5, 2)
B
@(1, 1)
(4, 1)

1
2
1C
C
3C
C
2A
1

9/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

NMF
A = WH, A Wk Hk
Factorization not unique:

(via Chih-Jen Lins NMF solver)

1
0
1.2850
B 0.4711 0.8065C
B
C
C
Wk = B
B 8.4380 0.0365C
@ 0.0217 6.7563A
10.8476
0

0.7968
0
0.0928
Hk =
0
2.9321 5.9337
0

1.0214
0.2885

0.0567
0.1667

Clustering and Ranking:

0

Rr =

(1, 2)
B(2, 2)
B
B(4, 2)
B
@(3, 1)
(5, 1)

1
2
3C
C
1C
C
2A
1

Rc =

(2, 2)
B(3, 2)
B
B(5, 2)
B
@(1, 1)
(4, 1)

1
2
1C
C
3C
C
2A
1

Order of k does not preserve the ranking of cluster importance

MMDS08

9/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Sparse Nonnegative Factorization

MMDS08

10/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Sparse NMF
A WH,

W , H 0,

low rank and sparse

Kim and Park (2007)

1
kA
W ,H0 2

min

MMDS08

WHk2F + kW k2F +

khj k12

11/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Sparse NMF
A WH,

W , H 0,

low rank and sparse

Kim and Park (2007)

1
kA
W ,H0 2

min

WHk2F + kW k2F +

khj k12

Alternating nonnegative least squares (NLS) on

 T 
 T  2
H
A
T
,

min
W
I
0 F
W 0



  2

W
A


min T H
0
e
H0

Sparse H
MMDS08

11/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Sparse NMF via BPDN

1
kA
W ,H0 2

min

MMDS08

WHk2F +

kwi k1 +

khj k1

12/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Sparse NMF via BPDN

1
kA
W ,H0 2

min

WHk2F +

kwi k1 +

khj k1

Alternating BPDN on
AT k2 +

kwi k1

min 12 kW H Ak2 +

khj k1

1
kH TW T
W 0 2

min

H0

Sparse W and H

MMDS08

12/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

L12 or L1?
Kim and Park (2007):
min 12 kAx bk2 + kxk21
x0

x 0 only as
(Nevertheless, Kim and Park report sparse solutions with moderate )

MMDS08

13/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

L12 or L1?
Kim and Park (2007):
min 12 kAx bk2 + kxk21
x0

x 0 only as
(Nevertheless, Kim and Park report sparse solutions with moderate )

BPDN:
min 12 kAx bk2 + kxk1
x0

x = 0 for kATbk
x very sparse for = 0.9kATbk say
Easy to control the sparsity of x
MMDS08

13/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

SNMF via BPDN implementation

1
kA
U,D,V 0 2

min

UDV T k2F +

i kDui k1 +

j kDvj k1

Alternating BPDN on

MMDS08

min 1 kUvj
vj 0 2

a.j k2 + j kvj k1 ,

normalize V VD

min 1 kV ui
ui 0 2

ai. k2 + i kui k1 ,

normalize U UD

14/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

SNMF via BPDN implementation

1
kA
U,D,V 0 2

min

UDV T k2F +

i kDui k1 +

j kDvj k1

Alternating BPDN on
min 1 kUvj
vj 0 2

a.j k2 + j kvj k1 ,

normalize V VD

min 1 kV ui
ui 0 2

ai. k2 + i kui k1 ,

normalize U UD

j kU Ta.j k
i kV Tai. k

MMDS08

= sparsity input parameter

= 0.9 or 0.8 say

14/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

SNMF via BPDN implementation

1
kA
U,D,V 0 2

min

UDV T k2F +

i kDui k1 +

j kDvj k1

Alternating BPDN on
min 1 kUvj
vj 0 2

a.j k2 + j kvj k1 ,

normalize V VD

min 1 kV ui
ui 0 2

ai. k2 + i kui k1 ,

normalize U UD

j kU Ta.j k
i kV Tai. k
At some point, freeze D
MMDS08

= sparsity input parameter

= 0.9 or 0.8 say
(Also j i stop changing)
14/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

BPDN solvers

MMDS08

15/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

BPDN solvers

min kxk1 + 12 kAx bk2

l1 ls

Kim, Koh, Lustig, Boyd et al 2007

GPSR

Figueiredo, Nowak & Wright 2007

SPGL1

van den Berg & Friedlander 2007

BPdual

Friedlander & Saunders 2007

Greedy
Interior, CG
Interior, LSQR
Orthogonal blocks
Active-set, all
Active-set, all
Double greedy
Primal barrier, PCG
Gradient Projection
Spectral GP, all
Active-set on dual

LPdual

Friedlander & Saunders 2007

Active-set on dual, x 0

lsNMF, lsNTF

Friedlander & Hatz 2007

Sparse NMF and NTF

OMP

Davis, Mallat et al 1997

BPDN-interior

Chen, Donoho & S, 1998, 2001

PDSCO, PDCO

Saunders 1997, 2002

BCR

Sardy, Bruce & Tseng 2000

Homotopy

Osborne et al 2000

LARS

Efron, Hastie, Tibshirani 2004

STOMP

Donoho, Tsaig, et al 2006

(BCLS subproblem solver)

MMDS08

16/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

LPdual solver
Active-set method for dual of regularized LP:
min e Tx + 12 ky k2
x, y

st

Ax + y = b,

miny b Ty + 12 ky k2

MMDS08

st

x 0

ATy e

17/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

LPdual solver
Active-set method for dual of regularized LP:
min e Tx + 12 ky k2
x, y

st

Ax + y = b,

miny b Ty + 12 ky k2

st

x 0

ATy e

B columns of A for active constraints (B Ty = e)

Initially y = 0, B empty
Selects columns of B in mostly greedy manner

MMDS08

17/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

LPdual solver
Active-set method for dual of regularized LP:
min e Tx + 12 ky k2
x, y

st

Ax + y = b,

miny b Ty + 12 ky k2

st

x 0

ATy e

B columns of A for active constraints (B Ty = e)

Initially y = 0, B empty
Selects columns of B in mostly greedy manner
Main work per iteration:
Solve min kBx g k
Form dy = (g Bx)/
Form dz = ATdy
Add or delete a column of B
MMDS08

17/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

SNMF Results

MMDS08

18/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Sparse NMF example

Sparse solution: k = 2

Uk =

Dk =

MMDS08

C
C
0.6146C
C
A
0.7889

0.1859
B0.1170
B
B
B
@0.9756
27.51

0
B0.4428
B

Vk = B
B0.8963
@
0.0253

1
0.6153
C
C
C
C
0.7877A
0.0303

10.69

19/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Sparse NMF example

Sparse solution: k = 2

Uk =

Dk =

C
C
0.6146C
C
A
0.7889

0.1859
B0.1170
B
B
B
@0.9756
27.51

MMDS08

B0.4428
B

Vk = B
B0.8963
@
0.0253

1
0.6153
C
C
C
C
0.7877A
0.0303

10.69

Clustering
0 and Ranking:
1
Rr =

(1, 1)
B(2, 1)
B
B(4, 1)
B
@(3, 2)
(5, 2)

2
3C
C
1C
C
2A
1

Rc =

(2, 1)
B(3, 1)
B
B(1, 2)
B
@(4, 2)
(5, 2)

1
2
1C
C
2C
C
1A
3

19/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Sparse NMF example

Sparse solution: k = 2

Uk =

Dk =

C
C
0.6146C
C
A
0.7889

0.1859
B0.1170
B
B
B
@0.9756
27.51

B0.4428
B

Vk = B
B0.8963
@
0.0253

1
0.6153
C
C
C
C
0.7877A
0.0303

10.69

Clustering
0 and Ranking:
1
Rr =

(1, 1)
B(2, 1)
B
B(4, 1)
B
@(3, 2)
(5, 2)

2
3C
C
1C
C
2A
1

Rc =

(2, 1)
B(3, 1)
B
B(1, 2)
B
@(4, 2)
(5, 2)

1
2
1C
C
2C
C
1A
3

Order of k does preserve the ranking of cluster importance

MMDS08

19/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

m = n = 450, k = 200, increasing sparsity

Alternating BPDN iterations

Time

90

3500

80

3000

70

2500

60
2000
50
1500
40
1000

30

500

20
10
0.2

0.4

0.6

0.8

0
0.2

0.4

0.6

0.8

0.8

||A UDVT||F / ||A||F

Nonzeros per row of U and V

20

0.4
0.35

15

0.3
0.25

10

0.2
0.15

0.1
0.05

0
0.2

MMDS08

0.4

0.6

0.8

0
0.2

0.4

0.6

20/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Real Application Examples

MMDS08

21/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Keyword clusterings
About 8000 stem terms

MMDS08

22/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Keyword clusterings
About 8000 stem terms
Create term similarity matrix A

MMDS08

22/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Keyword clusterings
About 8000 stem terms
Create term similarity matrix A
Sample clusters:
googladword
adword
googl
googlanalyt
yahoo
searchmarket
omnitur
msn
webtrend
adbrit
MMDS08

c++
cc++
java
c++java
c++program
c++unix
pascal
c++develop
c++programm
javaprogram
22/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

User input standardization

Field-of-study user input, about 400k unique entries
Cluster user inputs automatically
Sample clusters:
Abbreviations, variation of the same word or typos
hr, human resources, hrm
film production, film, theatre, acting, theater

New words
physical therapy, kinesiology

Similar disciplines
materials science and engineering, materials science, materials
engineering

Foreign language
business economics, bedrijfseconomie
bedrijfskundige informatica, business informatics, informatica

Noise elimination, or crowded cluster

business administration, business, mba, project management,
master in business administration, business administration,
master of business administration, technology, business admin,
general education
MMDS08

23/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Thanks
Michael Friedlander (BP solvers)

MMDS08

24/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Thanks
Michael Friedlander (BP solvers)
Sou-Cheng Choi
Lek-Heng Lim

MMDS08

24/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Thanks
Michael Friedlander (BP solvers)
Sou-Cheng Choi
Lek-Heng Lim
Jay Kreps
Jonathan Goldman
Huitao Luo

MMDS08

24/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Thanks
Michael Friedlander (BP solvers)
Sou-Cheng Choi
Lek-Heng Lim
Jay Kreps
Jonathan Goldman
Huitao Luo
Understanding Complex Datasets
Data Mining with Matrix Decompositions

(useful book by David Skillicorn)

MMDS08

24/24

Introduction

SNMF motivation

Sparse NMF

BPDN solvers

SNMF results

Application examples

Thanks
Michael Friedlander (BP solvers)
Sou-Cheng Choi
Lek-Heng Lim
Jay Kreps
Jonathan Goldman
Huitao Luo
Understanding Complex Datasets
Data Mining with Matrix Decompositions

(useful book by David Skillicorn)

Michael Saunders
MMDS08

24/24