Least Squares Support Vector Machines Classifiers :

a multi two-spiral benchmark problem

Lukas, J.A. Su!kens, J. Van"ewalle

#ept. of $lectrical $n%ineerin% $SA&-S'S&A
atholieke (ni)ersiteit Leu)en
asteelpark Arenber% *+, ,--++* Leu)en, ,el%ium.
email : lukas.lukas.esat.kuleu)en.ac.be
ke!wor"s : Support Vector Machines, least squares SVM, pattern reco%nition, classifier.
Abstract
Classification problems ha)e arisen in man! applications, attractin% man! researches to "e)elop a")ance" classifier techniques.
A metho" calle" Support Vector Machines /SVM0 for pattern reco%nition an" function estimation has been intro"uce" b! Vapnik
/*1120 in the framework of statistical learnin% theor!. Since then there is a %rowin% interest on this kernel metho" for its
interestin% features. 'n this paper, a least squares )ersion /LS-SVM0 is e3plaine". LS-SVM e3presses the trainin% in terms of
sol)in% a set of linear equations instea" of qua"ratic pro%rammin% as for the stan"ar" SVM case. 'terati)e trainin% al%orithm for
LS-SVM base" on a con4u%ate %ra"ient metho" is then applie". &he LS-SVM is then able to sol)e lar%e scale classification
problems which is illustrate" on a multi two-spiral benchmark problem.
INTRODUCTION
Support Vector Machines /SVM0 for sol)in% pattern reco%nition an" nonlinear function estimation problems ha)e
been intro"uce" in 567. &he i"ea of SVM is mappin% the trainin% "ata nonlinearl! into a hi%her-"imensional feature
space, then construct a separatin% h!perplane with ma3imum mar%in there. &his !iel"s a nonlinear "ecision
boun"ar! in input space. ,! the use of a kernel function, either pol!nomial, splines, ra"ial basis function /8,90 or
multila!er perceptron, it is possible to compute the separatin% h!perplane without e3plicitl! carr!in% out the map
into the feature space. :hile classical ;eural ;etworks techniques suffer from the e3istence of man! local minima,
SVM solutions are obtaine" from qua"ratic pro%rammin% problems possessin% a %lobal solution.
8ecentl!, least squares /LS0 )ersions of SVM ha)e been in)esti%ate" for classification 527 an" function estimation
5<7. 'n these LS-SVM formulations one computes the solution b! sol)in% a linear s!stem instea" of qua"ratic
pro%rammin%. &his is "ue to the use of equalit! instea" of inequalit! constraints in the problem formulation. 'n 5*, =7
such linear s!stems ha)e been calle" arush-uhn-&ucker /&0 s!stems an" their numerical stabilit! has been
in)esti%ate". &his linear s!stem can be efficientl! sol)e" b! iterati)e metho"s such as con4u%ate %ra"ient 5>7, an"
enables sol)in% lar%e scale classification problems. As an e3ample we show the e3cellent performance on a multi
two-spiral benchmark problem, which is known to be a "ifficult test case for neural network classifiers 5-7.
LEAST SQUARES SUPPORT VECTOR MACHINES
?i)en a trainin% set of ; "ata points
N
k k k
x y
*
@ , A

, where
n
k
R x is the k-th input pattern an" R y
k
is the k-
th output pattern, the classifier can be constructe" usin% the support )ector metho" in the form
1
]
1

N
k
k k k
b x x K y sign x y
*
0 , / 0 /
where
k
are calle" support )alues an" b is a constant. &he
( ) , K
is the kernel, which can be either
( ) x x x x K
T
k k
, /linear SVM0B ( )
d T
k k
x x x x K 0 * / , + /pol!nomial SVM of "e%ree "0B
( ) 7 tanh5 , + x x x x K
T
k k
/multila!er perceptron SVM0, or ( ) @ C e3pA ,
>
>
>

k k
x x x x K /8,9
SVM0, where , an" are constants.
9or instance, the problem of classif!in% two classes is "efine" as
*
*
* 0 /
* 0 /

+

'

+
+ +
k
k
k
T
k
T
y
y
if
if
b x w
b x w

&his can also be written as

N k b x w y
k
T
k
,..., * , * 7 0 / 5 +
where ( ) is a nonlinear function mappin% of the input space to a hi%her "imensional space. LS-SVM classifiersD

+
N
k
k
T
LS
e b w
e w w e b w J
*
>
>
*
>
*
, ,
0 , , / min
sub4ects to the equalit! constraints
N k e b x w y
k k
T
k
,..., * , * 7 0 / 5 +
&he La%ran%ian is "efine" as
{ }

+ +
N
k
k k
T
k k LS
e b x w y J e b w L
*
* 7 0 / 5 0 B , , /
with La%ran%e multipliers R
k
/calle" support )alues0.
&he con"itions for optimalit! are %i)en b!

'

+ +

+ * 7 0 / 5 +
+
+ +
0 / +
*
*
k k
T
k
L
k k e
L
N
k
k k b
L
N
k
k k k w
L
e b x w y
e
y
x y w
k
k

for N k ,..., * . After elimination of w an" e one obtains the solution

1
]
1

1
]
1

1
]
1

+

v
T
T
b
I ZZ Y
Y
*
+ +
*

with 7 B...B 5 7, * B...B * 5 * 7, B...B 5 7, 0 / B...B 0 / 5
* * * * N v N N
T
N
T
e e e y y Y y x y x Z an"
7 B...B 5
* N
. MercerEs con"ition is applie" to the matri3
T
ZZ
with
0 , /
0 / 0 /
l k l k
l
T
k l k kl
x x K y y
x x y y

&he kernel parameters, i.e. for 8,9 kernel, can be optimall! chosen b! optimiFin% an upper boun" on the VC
"imension. &he support )alues k are proportional to the errors at the "ata points in the LS-SVM case, while in the
stan"ar" SVM case man! support )alues are t!picall! equal to Fero. :hen sol)in% lar%e linear s!stems, it becomes
nee"e" to appl! iterati)e metho"s 5>7.
BENCHMARK : MULTI TWO-SPIRAL PROBLEM
Gne of the well-known benchmark problems for assessin% the qualit! of neural networks classifiers is two-spiral
problem 5-7. 'n 527 the e3cellent trainin% an" %eneraliFation performance of LS-SVM with 8,9 kernel on this
problem has been shown. 'n the followin% a more complicate" multi two-spiral classification problem is "epicte" in
9i%ure *. ?i)en are =-> trainin% "ata which consist of the two classes as in"icate" b! HIH an" HoH. &he LS-SVM
classifier usin% 8,9 kernel with J * an" J 2+. &he resultin% classifier with support )alues k an" bias term b
obtaine" from the lar%e scale al%orithm is shown on 9i%ure *. &akin% =-> support )alues one has no
misclassification on the trainin% set, to%ether with e3cellent %eneraliFation as is clear from the "ecision boun"ar!
between the black an" white re%ions. &he support )alue spectrum is "epicte" in the 9i%ure >, here the obtaine"
support )alues are sorte" from lar%est to smallest.
FIGURE . Multi two-spiral classification problem with =-> trainin% "ata /class * an" class > are in"icate" b! KIE an" KoE0. 9rom
black an" white re%ions which "etermine the "ecision boun"ar! between two classes is clearl! shown the e3cellent %eneraliFation
performance of the LS-SVM with 8,9 kernel.
FIGURE !" &he spectrum of support )alues relate" to the classification problem in 9i%ure *. &he support )alues k are sorte"
from the lar%est to the smallest )alue for the %i)en trainin% "ata set.
CONCLUSION
'n this paper a least squares )ersion of Support Vector Machines /LS-SVM0 is e3plaine". &he solution of the linear
s!stem can be calculate" efficientl! usin% a con4u%ate %ra"ient metho". As illustrate" in the "ifficult multi two-spiral
classification problem, e3cellent %eneraliFation performance can be obtaine" usin% LS-SVM approach in separable
case. &he performance of the classifier turns out to be quite robust with respect to tunin% parameters of the
al%orithm.
ACKNOWLEDGEMENTS
&his research work was carrie" out at the $SA& laborator! an" the 'nter"isciplinar! Center of ;eural ;etworks
'C;; of the atholieke (ni)ersiteit Leu)en, in the framework of the 9:G pro4ect Learnin% an" GptimiFation : an
'nter"isciplinar! Approach, the ,el%ian Lro%ram on 'nteruni)ersit! Loles of Attraction, initiate" b! the ,el%ian
State, Lrime MinisterHs Gffice for Science, &echnolo%! an" Culture /'(AL L=-+> M '(AL L=->=0 an" the Concerte"
Action Lro4ect M$9'S&G of the 9lemish Communit!. Johan Su!kens is a post"octoral researcher with the ;ational
9un" for Scientific 8esearch 9:G - 9lan"ers.
REFERENCES
5*7 9letcher 8., Johnson &., NGn the stabilit! of null-space metho" for & s!stemsO, SIAM J. Matrix Anal. Al.,
Vol. #, ;o. =, *116, pp. 1-P-12P.
5>7 ?olub ?.Q., Van Loan C.9., Matrix !"#$tati"ns, ,altimore M#, John Qopkins (ni)ersit! Lress, *1P1.
5-7 8i"ella S., 8o)etta S., Runino 8., NCircular backpropa%ation networks for classificationO, I%%% Transa&ti"ns "n
Ne$ral Netw"rks, Vol. #, ;o. *, *116, pp. P=-16.
5=7 Sun J.-?., NStructure" backwar"s errors for & s!stemsO, Linear Algebra and its Ali&ati"ns, >PP, *111,
pp. 62-PP.
527 Su!kens J.A.., Van"ewalle J., NLeast squares support )ector machine classifiersO, Ne$ral 'r"&essing Letter,
)ol. $, no. -, Jun. *111, pp. >1---++.
5<7 Su!kens J.A.., Van"ewalle J., N&rainin% multila!er perceptron classifiers base" on a mo"ifie" support )ector
metho"O, I%%% Transa&ti"ns "n Ne$ral Netw"rks, *111.
567 Vapnik V. T(e Nat$re "f Statisti&al Learning T(e"ry. Sprin%er-Verla%, ;ew Sork, *112.