Tutorial
Jason Weston
NEC Labs America
4 Independence Way, Princeton, USA.
jasonw@neclabs.com
1 Support Vector Machines: history
• SVMs introduced in COLT92 by Boser, Guyon & Vapnik. Became
rather popular since.
    margin

margin
+ +
+
+ +
Nice properties: convex, theoretically motivated, nonlinear with kernels..
4 Preliminaries:
• Machine learning is about learning structure from data.
vs.
1 X
m
Remp (α) = `(f (xi , α), yi ) = Training Error
m i=1
s
h(log( 2m + 1) − log( 4)
η
R(α) ≤ Remp (α) + h
m
where h is the VC dimension of the set of functions parameterized by α.
• The VC dimension of a set of functions is a measure of their capacity
or complexity.
• If you can describe a lot of different phenomena with a set of
functions then the value of h is large.
[VC dim = the maximum number of points that can be separated in all
possible ways by that set of functions.]
10 VC dimension:
Simplification of bound:
Test Error ≤ Training Error + Complexity of set of Models
• If you take a high capacity set of functions (explain a lot) you get low
training error. But you might ”overfit”.
• If you take a very simple set of models, you have low complexity, but
won’t get low training error.
12 Capacity of a set of functions (classification)
hyperplane fit
true function
x
14 Controlling the risk: model complexity
Confidence interval
Empirical risk
(training error)
h1 h* hn
S1 S* Sn
15 Capacity of hyperplanes
h ≤ R2 A2 .
where R is the radius of the smallest sphere around the origin containing
X ∗.
=⇒ minimize w2 and have low capacity
=⇒ minimizing w2 equivalent to obtaining a large margin classifier
<w, x> + b > 0
● ◆
●
◆
<w, x> + b < 0 w ◆
●
●
●
{x  <w, x> + b = 0}
{x  <w, x> + b = +1}
{x  <w, x> + b = −1} Note:
◆
<w, x1> + b = +1
❍ ◆ x1 yi = +1 <w, x2> + b = −1
x2❍
=> <w , (x1−x2)> = 2
◆
, w
yi = −1 w ◆ => < , (x1−x2) = 2
>
w w
❍
❍
❍
{x  <w, x> + b = 0}
16 Linear Support Vector Machines (at last!)
So, we would like to find the function which minimizes an objective like:
Training Error + Complexity term
We write that as:
1 X
m
`(f (xi , α), yi ) + Complexity term
m i=1
For now we will choose the set of hyperplanes (we will extend this later),
so f (x) = (w · x) + b:
1 X
m
`(w · xi + b, yi ) + w2
m i=1
subject to mini w · xi  = 1.
17 Linear Support Vector Machines II
That function before was a little difficult to minimize because of the step
function in `(y, ŷ) (either 1 or 0).
Let’s assume we can separate the data perfectly. Then we can optimize
the following:
Minimize w2 , subject to:
(w · xi + b) ≥ 1, if yi = 1
(w · xi + b) ≤ −1, if yi = −1
The last two constraints can be compacted to:
yi (w · xi + b) ≥ 1
yi (w · xi + b) ≥ 1 − ξi , ξi ≥ 0
1 X
m
`(w · xi + b, yi ) + w2
m i=1
margin
+ +
+ ξi
+  +

19 Support Vector Machines  Primal
• Decision function:
f (x) = w · x + b
• Primal formulation:
1 X
min P (w, b) = kwk2 + C H1 [ yi f (xi ) ]
2 {z } i
 {z }
maximize margin
minimize training error
Ideally H1 would count the number of errors, approximate with:
H1(z)
0 z
20 SVMs : nonlinear case
Linear classifiers aren’t complex enough sometimes. SVM solution:
Map data into a richer feature space including nonlinear features, then
construct a hyperplane in that space so all other equations are the same!
Formally, preprocess the data with:
x 7→ Φ(x)
f (x) = w · Φ(x) + b.
21 SVMs : polynomial mapping
Φ : R2 → R3
p
(x1 , x2 ) 7→ (z1 , z2 , z3 ) := (x1 , (2)x1 x2 , x22 )
2
x2
z3
✕
✕ ✕ ✕
✕ ✕
✕ ✕
✕ ✕
✕ ✕
✕
✕
✕ ✕ ❍ ✕ ✕
✕
❍
❍
❍
x1 ❍
❍ ✕
❍
❍ ✕
✕
❍
❍
❍❍ ✕
✕
z1
✕ ❍ ✕ ❍ ❍
❍
✕ ✕
✕
✕
✕ ✕ ✕
✕ z2
22 SVMs : nonlinear case II
For example MNIST handwriting recognition.
60,000 training examples, 10000 test examples, 28x28.
Linear SVM has around 8.5% test error.
Polynomial SVM has around 1% test error.
5 0 4 1 9 2 1 3 1 4
3 5 3 6 1 7 2 8 6 9
4 0 9 1 1 2 4 3 2 7
3 8 6 9 0 5 6 0 7 6
1 8 7 9 3 9 8 5 9 3
3 0 7 4 9 8 0 9 4 1
4 4 6 0 4 5 6 1 0 0
1 7 1 6 3 0 2 1 1 7
9 0 2 6 7 8 3 9 0 4
6 7 4 6 8 0 7 8 3 1
23 SVMs : full MNIST results
We can rewrite all the SVM equations we saw before, but with the
Pm
w = i=1 αi Φ(xi ) equation:
• Decision function:
X
f (x) = αi Φ(xi ) · Φ(x) + b
i
X
= αi K(xi , x) + b
i
• Dual formulation:
1 X X
m
2
min P (w, b) = k αi Φ(xi )k + C H1 [ yi f (xi ) ]
2 i=1 i
 {z }  {z }
maximize margin minimize training error
26 Support Vector Machines  Dual
But people normally write it like this:
• Dual formulation:
1X X P α =0
i i
min D(α) = αi αj Φ(xi )·Φ(xj )− yi αi s.t.
α 2 i,j 0≤yi αi ≤C
i
2 2
= x21 x0 1 + 2x1 x0 1 x2 x0 2 + x22 x0 2
is the same as:
. .
x x' Φ(x) Φ(x')
There is much more in the field of SVMs/ kernel machines than we could
cover here, including:
Please see text books such as the ones by Cristianini & ShaweTaylor or
by Schoelkopf and Smola.
30 SVMs : software
• SVMLight (C)
As well as complete machine learning toolboxes that include SVMs:
• Torch (C++)
• Spider (Matlab)
• Weka (Java)
All available through www.kernelmachines.org.