Professional Documents
Culture Documents
=================
-
Introduction
Installation
Quick Start
Design Description
Data Structures
Utility Functions
Additional Information
Introduction
============
Python (http://www.python.org/) is a programming language suitable for
rapid
development. This tool provides a simple Python interface to LIBSVM, a
library
for support vector machines (http://www.csie.ntu.edu.tw/~cjlin/libsvm).
The
interface is very easy to use as the usage is the same as that of LIBSVM.
The
interface is developed with the built-in Python library "ctypes."
Installation
============
On Unix systems, type
> make
The interface needs only LIBSVM shared library, which is generated by
the above command. We assume that the shared library is on the LIBSVM
main directory or in the system path.
For windows, the shared library libsvm.dll for 32-bit python is ready
in the directory `..\windows'. You can also copy it to the system
directory (e.g., `C:\WINDOWS\system32\' for Windows XP). To regenerate
the shared library, please follow the instruction of building windows
binaries in LIBSVM README.
Quick Start
===========
There are two levels of usage. The high-level one uses utility functions
in svmutil.py and the usage is the same as the LIBSVM MATLAB interface.
>>> from svmutil import *
# Read data in LIBSVM format
>>> y, x = svm_read_problem('../heart_scale')
- class svm_model:
There are two ways to obtain an instance of svm_model:
>>> model = svm_train(y, x)
>>> model = svm_load_model('model_file_name')
Note that the returned structure of interface functions
libsvm.svm_train and libsvm.svm_load_model is a ctypes pointer of
svm_model, which is different from the svm_model object returned
by svm_train and svm_load_model in svmutil.py. We provide a
function toPyModel for the conversion:
>>> model_ptr = libsvm.svm_train(prob, param)
>>> model = toPyModel(model_ptr)
If you obtain a model in a way other than the above approaches,
handle it carefully to avoid memory leak or segmentation fault.
Some interface functions to access LIBSVM models are wrapped as
members of the class svm_model:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
svm_type = model.get_svm_type()
nr_class = model.get_nr_class()
svr_probability = model.get_svr_probability()
class_labels = model.get_labels()
sv_indices = model.get_sv_indices()
nr_sv = model.get_nr_sv()
is_prob_model = model.is_probability_model()
support_vector_coefficients = model.get_sv_coef()
support_vectors = model.get_SV()
Utility Functions
=================
To use utility functions, type
>>> from svmutil import *
The above command loads
svm_train()
:
svm_predict()
:
svm_read_problem() :
svm_load_model()
:
svm_save_model()
:
evaluations()
:
- Function: svm_train
There are three ways to call svm_train()
>>> model = svm_train(y, x [, 'training_options'])
>>> model = svm_train(prob [, 'training_options'])
>>> model = svm_train(prob, param)
y, x = svm_read_problem('../heart_scale')
prob = svm_problem(y, x)
param = svm_parameter('-s 3 -c 5 -h 0')
m = svm_train(y, x, '-c 5')
m = svm_train(prob, '-t 2 -c 5')
m = svm_train(prob, param)
CV_ACC = svm_train(y, x, '-v 3')
- Function: svm_predict
To predict testing data with a model, use
>>> p_labs, p_acc, p_vals = svm_predict(y, x, model
[,'predicting_options'])
y: a list/tuple of l true labels (type must be int/double). It is
used
for calculating the accuracy. Use [0]*len(x) if true labels are
unavailable.
x: a list/tuple of l predicting instances. The feature vector of
each predicting instance is an instance of list/tuple or
dictionary.
svm_read_problem/svm_load_model/svm_save_model
You must have libsvm and gnuplot installed before using it. The package
gnuplot is available at http://www.gnuplot.info/
On Mac OSX, the precompiled gnuplot file needs the library Aquarterm,
which thus must be installed as well. In addition, this version of
gnuplot does not support png, so you need to change "set term png
transparent small" and use other image formats. For example, you may
have "set term pbm small color".
Usage: grid.py [grid_options] [svm_options] dataset
grid_options :
-log2c {begin,end,step | "null"} : set the range of c (default -5,15,2)
begin,end,step -- c_range = 2^{begin,...,begin+k*step,...,end}
"null"
-- do not grid with c
-log2g {begin,end,step | "null"} : set the range of g (default 3,-15,-2)
begin,end,step -- g_range = 2^{begin,...,begin+k*step,...,end}
"null"
-- do not grid with g
-v n : n-fold cross validation (default 5)
-svmtrain pathname : set svm executable path and name
-gnuplot {pathname | "null"} :
pathname -- set gnuplot executable path and name
"null"
-- do not plot
-out {pathname | "null"} : (default dataset.out)
pathname -- set output file path and name
"null"
-- do not output file
-png pathname : set graphic output file path and name (default
dataset.png)
-resume [pathname] : resume the grid task using an existing output file
(default pathname is dataset.out)
Use this option only if some parameters have been checked for the
SAME data.
svm_options : additional options for svm-train
The program conducts v-fold cross validation using parameter C (and
gamma)
= 2^begin, 2^(begin+step), ..., 2^end.
You can specify where the libsvm executable and gnuplot are using the
-svmtrain and -gnuplot parameters.
For windows users, please use pgnuplot.exe. If you are using gnuplot
3.7.1, please upgrade to version 3.7.3 or higher. The version 3.7.1
has a bug. If you use cygwin on windows, please use gunplot-x11.
If the task is terminated accidentally or you would like to change the
range of parameters, you can apply '-resume' to save time by re-using
previous results. You may specify the output file of a previous run
or use the default (i.e., dataset.out) without giving a name. Please
note that the same condition must be used in two runs. For example,
you cannot use '-v 10' earlier and resume the task with '-v 5'.
The value of some options can be "null." For example, `-log2c -1,0,1
each data point has one label (the color) that must be 1, 2,
or 3 and two attributes (x-axis and y-axis values) in
[0,1). For regression, each data point has one target value
(y-axis) and one attribute (x-axis values) in [0, 1).
Type `make' in respective directories to build them.
You need Qt library to build the Qt version.
(available from http://www.trolltech.com)
You need GTK+ library to build the GTK version.
(available from http://www.gtk.org)
The pre-built Windows binaries are in the `windows'
directory. We use Visual C++ on a 32-bit machine, so the
maximal cache size is 2GB.
`svm-train' Usage
=================
Usage: svm-train [options] training_set_file [model_file]
options:
-s svm_type : set type of SVM (default 0)
0 -- C-SVC
(multi-class classification)
1 -- nu-SVC
(multi-class classification)
2 -- one-class SVM
3 -- epsilon-SVR (regression)
4 -- nu-SVR
(regression)
-t kernel_type : set type of kernel function (default 2)
0 -- linear: u'*v
1 -- polynomial: (gamma*u'*v + coef0)^degree
2 -- radial basis function: exp(-gamma*|u-v|^2)
3 -- sigmoid: tanh(gamma*u'*v + coef0)
4 -- precomputed kernel (kernel values in training_set_file)
-d degree : set degree in kernel function (default 3)
-g gamma : set gamma in kernel function (default 1/num_features)
-r coef0 : set coef0 in kernel function (default 0)
-c cost : set the parameter C of C-SVC, epsilon-SVR, and nu-SVR (default
1)
-n nu : set the parameter nu of nu-SVC, one-class SVM, and nu-SVR
(default 0.5)
-p epsilon : set the epsilon in loss function of epsilon-SVR (default
0.1)
-m cachesize : set cache memory size in MB (default 100)
-e epsilon : set tolerance of termination criterion (default 0.001)
-h shrinking : whether to use the shrinking heuristics, 0 or 1 (default
1)
-b probability_estimates : whether to train a SVC or SVR model for
probability estimates, 0 or 1 (default 0)
-wi weight : set the parameter C of class i to weight*C, for C-SVC
(default 1)
-v n: n-fold cross validation mode
-q : quiet mode (no outputs)
The k in the -g option means the number of attributes in the input data.
option -v randomly splits the data into n parts and calculates cross
validation accuracy/mean squared error on them.
See libsvm FAQ for the meaning of outputs.
`svm-predict' Usage
===================
Usage: svm-predict [options] test_file model_file output_file
options:
-b probability_estimates: whether to predict probability estimates, 0 or
1 (default 0); for one-class SVM only 0 is supported
model_file is the model file generated by svm-train.
test_file is the test data you want to predict.
svm-predict will produce output in the output_file.
`svm-scale' Usage
=================
Usage: svm-scale [options] data_filename
options:
-l lower : x scaling lower limit (default -1)
-u upper : x scaling upper limit (default +1)
-y y_lower y_upper : y scaling limits (default: no y scaling)
-s save_filename : save scaling parameters to save_filename
-r restore_filename : restore scaling parameters from restore_filename
See 'Examples' in this file for examples.
Tips on Practical Use
=====================
* Scale your data. For example, scale each attribute to [0,1] or [-1,+1].
* For C-SVC, consider using the model selection tool in the tools
directory.
* nu in nu-SVC/one-class-SVM/nu-SVR approximates the fraction of training
errors and support vectors.
* If data for classification are unbalanced (e.g. many positive and
few negative), try different penalty parameters C by -wi (see
examples below).
* Specify larger cache size (i.e., larger -m) for huge problems.
Examples
========
> svm-scale -l -1 -u 1 -s range train > train.scale
> svm-scale -r range test > test.scale
Scale each feature of the training data to be in [-1,1]. Scaling
factors are stored in the file range and then used for scaling the
test data.
> svm-train -s 0 -c 5 -t 2 -g 0.5 -e 0.1 data_file
Train a classifier with RBF kernel exp(-0.5|u-v|^2), C=10, and
stopping tolerance 0.1.
> svm-train -s 3 -p 0.1 -t 0 data_file
Solve SVM regression with linear kernel u'v and epsilon=0.1
in the loss function.
> svm-train -c 10 -w1 1 -w-2 5 -w4 2 data_file
Train a classifier with penalty 10 = 1 * 10 for class 1, penalty 50 =
5 * 10 for class -2, and penalty 20 = 2 * 10 for class 4.
> svm-train -s 0 -c 100 -g 0.1 -v 5 data_file
Do five-fold cross validation for the classifier using
the parameters C = 100 and gamma = 0.1
> svm-train -s 0 -b 1 data_file
> svm-predict -b 1 test_file data_file.model output_file
Obtain a model with probability information and predict test data with
probability estimates
Precomputed Kernels
===================
Users may precompute kernel values and input them as training and
testing files. Then libsvm does not need the original
training/testing sets.
Assume there are L training instances x1, ..., xL and.
Let K(x, y) be the kernel
value of two instances x and y. The input formats
are:
New training instance for xi:
<label> 0:i 1:K(xi,x1) ... L:K(xi,xL)
New testing instance for any x:
<label> 0:? 1:K(x,x1) ... L:K(x,xL)
That is, in the training file the first column must be the "ID" of
xi. In testing, ? can be any value.
All kernel values including ZEROs must be explicitly provided. Any
permutation or random subsets of the training/testing files are also
valid (see examples below).
15
1:1
3:1
15
3:1
0
0
0
0.5
0.1
[
]
]
]
]
struct svm_parameter
{
int svm_type;
int kernel_type;
int degree; /* for poly */
double gamma;
/* for poly/rbf/sigmoid */
double coef0;
/* for poly/sigmoid */
/* these are for training only */
double cache_size; /* in MB */
double eps; /* stopping criteria */
double C; /* for C_SVC, EPSILON_SVR, and NU_SVR */
int nr_weight;
/* for C_SVC */
int *weight_label;
/* for C_SVC */
double* weight;
/* for C_SVC */
double nu; /* for NU_SVC, ONE_CLASS, and NU_SVR */
double p; /* for EPSILON_SVR */
int shrinking;
/* use the shrinking heuristics */
int probability; /* do probability estimates */
};
svm_type can be one of C_SVC, NU_SVC, ONE_CLASS, EPSILON_SVR, NU_SVR.
C_SVC:
NU_SVC:
ONE_CLASS:
EPSILON_SVR:
NU_SVR:
C-SVM classification
nu-SVM classification
one-class-SVM
epsilon-SVM regression
nu-SVM regression
};
param describes the parameters used to obtain the model.
nr_class is the number of classes. It is 2 for regression and oneclass SVM.
l is the number of support vectors. SV and sv_coef are support
vectors and the corresponding coefficients, respectively. Assume
there are
information, 0 is returned.
- Function: double svm_predict_values(const svm_model *model,
const svm_node *x, double* dec_values)
This function gives decision values on a test vector x given a
model, and return the predicted label (classification) or
the function value (regression).
For a classification model with nr_class classes, this function
gives nr_class*(nr_class-1)/2 decision values in the array
dec_values, where nr_class can be obtained from the function
svm_get_nr_class. The order is label[0] vs. label[1], ...,
label[0] vs. label[nr_class-1], label[1] vs. label[2], ...,
label[nr_class-2] vs. label[nr_class-1], where label can be
obtained from the function svm_get_labels. The returned value is
the predicted class for x. Note that when nr_class = 1, this
function does not give any decision value.
For a regression model, dec_values[0] and the returned value are
both the function value of x calculated using the model. For a
one-class model, dec_values[0] is the decision value of x, while
the returned value is +1/-1.
- Function: double svm_predict_probability(const struct svm_model *model,
const struct svm_node *x, double* prob_estimates);
This function does classification or regression on a test vector x
given a model with probability information.
For a classification model with probability information, this
function gives nr_class probability estimates in the array
prob_estimates. nr_class can be obtained from the function
svm_get_nr_class. The class with the highest probability is
returned. For regression/one-class SVM, the array prob_estimates
is unchanged and the returned value is the same as that of
svm_predict.
- Function: const char *svm_check_parameter(const struct svm_problem
*prob,
const struct svm_parameter
*param);
This function checks whether the parameters are within the feasible
range of the problem. This function should be called before calling
svm_train() and svm_cross_validation(). It returns NULL if the
parameters are feasible, otherwise an error message is returned.
- Function: int svm_check_probability_model(const struct svm_model
*model);
This function checks whether the model contains required
information to do probability estimates. If so, it returns
+1. Otherwise, 0 is returned. This function should be called
-classpath
-classpath
-classpath
-classpath
libsvm.jar
libsvm.jar
libsvm.jar
libsvm.jar
svm_train <arguments>
svm_predict <arguments>
svm_toy
svm_scale <arguments>
Note that you need Java 1.5 (5.0) or above to run it.
You may need to add Java runtime library (like classes.zip) to the
classpath.
You may need to increase maximum Java heap size.
1. Open a DOS command box (or Visual Studio Command Prompt) and change
to libsvm directory. If environment variables of VC++ have not been
set, type
"C:\Program Files\Microsoft Visual Studio 10.0\VC\bin\vcvars32.bat"
You may have to modify the above command according which version of
VC++ or where it is installed.
2. Type
nmake -f Makefile.win clean all
3. (optional) To build shared library libsvm.dll, type
nmake -f Makefile.win lib
4. (optional) To build 64-bit windows binaries, you must
(1) Setup vcvars64.bat instead of vcvars32.bat
(2) Change CFLAGS in Makefile.win: /D _WIN32 to /D _WIN64
Another way is to build them from Visual C++ environment. See details
in libsvm FAQ.
- Additional Tools: Sub-sampling, Parameter Selection, Format checking,
etc.
=========================================================================
===
See the README file in the tools directory.
MATLAB/OCTAVE Interface
=======================
Please check the file README in the directory `matlab'.
Python Interface
================
See the README file in python directory.
Additional Information
======================
If you find LIBSVM helpful, please cite it as
Chih-Chung Chang and Chih-Jen Lin, LIBSVM : a library for support
vector machines. ACM Transactions on Intelligent Systems and
Technology, 2:27:1--27:27, 2011. Software available at
http://www.csie.ntu.edu.tw/~cjlin/libsvm
LIBSVM implementation document is available at
http://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf
Libsvm is available at
http://www.csie.ntu.edu.tw/~cjlin/libsvm
Please read the COPYRIGHT file before using libsvm.
Table of Contents
=================
- Quick Start
- Installation and Data Format
- `svm-train' Usage
- `svm-predict' Usage
- `svm-scale' Usage
- Tips on Practical Use
- Examples
- Precomputed Kernels
- Library Usage
- Java Version
- Building Windows Binaries
- Additional Tools: Sub-sampling, Parameter Selection, Format checking, etc.
- MATLAB/OCTAVE Interface
- Python Interface
- Additional Information
Quick Start
===========
If you are new to SVM and if the data is not large, please go to
`tools' directory and use easy.py after installation. It does
everything automatic -- from data scaling to parameter selection.
Type `svm-train heart_scale', and the program will read the training
data and output the model file `heart_scale.model'. If you have a test
set called heart_scale.t, then type `svm-predict heart_scale.t
heart_scale.model output' to see the prediction accuracy. The `output'
For classification, if training data are in only one class (i.e., all
labels are the same), then `svm-train' issues a warning message:
`Warning: training data in only one class. See README for details,'
which means the training data is very unbalanced. The label in the
training data is directly returned when testing.
svm-scale:
svm-toy:
You can enter options in the bottom of the window, the syntax of
options is the same as `svm-train'.
Note that "load" and "save" consider dense data format both in
classification and the regression cases. For classification,
each data point has one label (the color) that must be 1, 2,
or 3 and two attributes (x-axis and y-axis values) in
[0,1). For regression, each data point has one target value
(y-axis) and one attribute (x-axis values) in [0, 1).
`svm-train' Usage
=================
0 -- C-SVC
(multi-class classification)
1 -- nu-SVC
(multi-class classification)
2 -- one-class SVM
3 -- epsilon-SVR
(regression)
4 -- nu-SVR
(regression)
The k in the -g option means the number of attributes in the input data.
option -v randomly splits the data into n parts and calculates cross
validation accuracy/mean squared error on them.
`svm-predict' Usage
===================
`svm-scale' Usage
=================
* Scale your data. For example, scale each attribute to [0,1] or [-1,+1].
* For C-SVC, consider using the model selection tool in the tools directory.
* nu in nu-SVC/one-class-SVM/nu-SVR approximates the fraction of training
errors and support vectors.
* If data for classification are unbalanced (e.g. many positive and
few negative), try different penalty parameters C by -wi (see
examples below).
* Specify larger cache size (i.e., larger -m) for huge problems.
Examples
========
factors are stored in the file range and then used for scaling the
test data.
Obtain a model with probability information and predict test data with
probability estimates
Precomputed Kernels
===================
Users may precompute kernel values and input them as training and
testing files. Then libsvm does not need the original
training/testing sets.
That is, in the training file the first column must be the "ID" of
xi. In testing, ? can be any value.
Examples:
2:3
25
15 1:1
4:3
3:1
3:1
Any subset of the above training file is also valid. For example,
Library Usage
=============
Before you classify test data, you need to construct an SVM model
(`svm_model') using training data. A model can also be saved in
a file for later use. Once an SVM model is available, you can use it
struct svm_problem
{
int l;
double *y;
struct svm_node **x;
};
where `l' is the number of training data, and `y' is an array containing
their target values. (integers in classification, real numbers in
regression) `x' is an array of pointers, each of which points to a sparse
representation (array of svm_node) of one training vector.
LABEL
-----
----- -----
0.1
0.2
0.1
0.3
-1.2
0.4
0.1
1.4
0.5
-0.1
-0.2
0.1
1.1
0.1
l=5
y -> 1 2 1 2 3
struct svm_node
{
int index;
double value;
};
index = -1 indicates the end of one vector. Note that indices must
be in ASCENDING order.
struct svm_parameter
{
int svm_type;
int kernel_type;
int degree;
/* for poly */
double gamma;
/* for poly/rbf/sigmoid */
double coef0;
/* for poly/sigmoid */
int nr_weight;
/* for C_SVC */
int *weight_label;
/* for C_SVC */
double* weight;
/* for C_SVC */
double nu;
double p;
/* for EPSILON_SVR */
C_SVC:
C-SVM classification
NU_SVC:
nu-SVM classification
ONE_CLASS:
one-class-SVM
EPSILON_SVR:
epsilon-SVM regression
NU_SVR:
nu-SVM regression
LINEAR:
u'*v
POLY:
(gamma*u'*v + coef0)^degree
RBF: exp(-gamma*|u-v|^2)
SIGMOID: tanh(gamma*u'*v + coef0)
PRECOMPUTED: kernel values in training_set_file
for some classes (If the weight for a class is not changed, it is
set to 1). This is useful for training classifier using unbalanced
input data or with asymmetric misclassification cost.
struct svm_model stores the model obtained from the training procedure.
It is not recommended to directly access entries in this structure.
Programmers should use the interface functions to get the values.
struct svm_model
{
struct svm_parameter param;
/* parameter */
int nr_class;
int l;
/* total #SV */
*/
/* SVs (SV[l]) */
double **sv_coef;
double *rho;
1][l]) */
double *probA;
double *probB;
int *sv_indices;
/* sv_indices[0,...,nSV-1] are values in
[1,...,num_traning_data] to indicate SVs in the training set */
int *label;
int *nSV;
/* XXX */
int free_sv;
};
+-+-+-+--------------------+
|1|1|1|
+-+-+-+--------------------+
|1|2|2|
+-+-+-+--------------------+
|1|2|3|
+-+-+-+--------------------+
|1|2|3|
+-+-+-+--------------------+
rho is the bias term (-b). probA and probB are parameters used in
probability outputs. If there are k classes, there are k*(k-1)/2
binary problems as well as rho, probA, and probB values. They are
aligned in the order of binary problems:
1 vs 2, 1 vs 3, ..., 1 vs k, 2 vs 3, ..., 2 vs k, ..., k-1 vs k.
This function outputs indices of support vectors into an array called sv_indices.
The size of sv_indices is the number of support vectors and can be obtained by calling
svm_get_nr_sv.
Each sv_indices[i] is in the range of [1, ..., num_traning_data].
information, 0 is returned.
This function checks whether the parameters are within the feasible
range of the problem. This function should be called before calling
svm_train() and svm_cross_validation(). It returns NULL if the
parameters are feasible, otherwise an error message is returned.
svm_predict_probability.
This function returns a pointer to the model read from the file,
or a null pointer if the model could not be loaded.
This function frees the memory used by the entries in a model structure.
This function frees the memory used by a model and destroys the model
structure. It is equivalent to svm_destroy_model, which
is deprecated after version 3.0.
Java Version
============
The pre-compiled java class archive `libsvm.jar' and its source files are
in the java directory. To run the programs, use
Note that you need Java 1.5 (5.0) or above to run it.
You may need to add Java runtime library (like classes.zip) to the classpath.
You may need to increase maximum Java heap size.
Library usages are similar to the C version. These functions are available:
1. Open a DOS command box (or Visual Studio Command Prompt) and change
to libsvm directory. If environment variables of VC++ have not been
set, type
You may have to modify the above command according which version of
VC++ or where it is installed.
2. Type
Another way is to build them from Visual C++ environment. See details
in libsvm FAQ.
MATLAB/OCTAVE Interface
=======================
Python Interface
================
Additional Information
======================
Acknowledgments:
This work was supported in part by the National Science
Council of Taiwan via the grant NSC 89-2213-E-002-013.
The authors thank their group members and users
for many helpful discussions and comments. They are listed in
http://www.csie.ntu.edu.tw/~cjlin/libsvm/acknowledgements
Introduction
============
options:
-s method : method of selection (default 0)
0 -- stratified selection (classification only)
1 -- random selection
Example
=======
Introduction
============
for cross validation and then draw contour of CV accuracy using gnuplot.
You must have libsvm and gnuplot installed before using it. The package
gnuplot is available at http://www.gnuplot.info/
On Mac OSX, the precompiled gnuplot file needs the library Aquarterm,
which thus must be installed as well. In addition, this version of
gnuplot does not support png, so you need to change "set term png
transparent small" and use other image formats. For example, you may
have "set term pbm small color".
grid_options :
-log2c {begin,end,step | "null"} : set the range of c (default -5,15,2)
begin,end,step -- c_range = 2^{begin,...,begin+k*step,...,end}
"null"
The program conducts v-fold cross validation using parameter C (and gamma)
= 2^begin, 2^(begin+step), ..., 2^end.
You can specify where the libsvm executable and gnuplot are using the
-svmtrain and -gnuplot parameters.
For windows users, please use pgnuplot.exe. If you are using gnuplot
3.7.1, please upgrade to version 3.7.3 or higher. The version 3.7.1
has a bug. If you use cygwin on windows, please use gunplot-x11.
The value of some options can be "null." For example, `-log2c -1,0,1
Example
=======
Users (in particular MS Windows users) may need to specify the path of
executable files. You can either change paths in the beginning of
grid.py or specify them in the command line. For example,
> grid.py -log2c -5,5,1 -svmtrain "c:\Program Files\libsvm\windows\svm-train.exe" gnuplot c:\tmp\gnuplot\binary\pgnuplot.exe -v 10 heart_scale
The following example saves running time by loading the output file of a previous run.
> python grid.py -log2c -7,7,1 -log2g -5,2,1 -v 5 -resume heart_scale.out heart_scale
and then setup your ssh so that the authentication works without
asking a password.
The same machine (e.g., linux5 here) can be listed more than once if
it has multiple CPUs or has more RAM. If the local machine is the
best, you can also enlarge the nr_local_worker. For example:
nr_local_worker = 2
Example:
If your system uses telnet instead of ssh, you list the computer names
in telnet_workers.
You need to specify `dataset' and `options' (default ''). See the following example.
> python
>>> rate
78.8889
>>> param
{'c': 0.5, 'g': 0.5}
Introduction
============
Example
=======
line 1: feature indices must be in an ascending order, previous/current features 3:1 2:4
Found 1 lines with error.
Introduction
SVMs (Support Vector Machines) are a useful technique for data classification. Although SVM is considered easier to use than Neural Networks, users not familiar with
it often get unsatisfactory results at first. Here we outline a cookbook approach
which usually gives reasonable results.
Note that this guide is not for SVM researchers nor do we guarantee you will
achieve the highest accuracy. Also, we do not intend to solve challenging or difficult problems. Our purpose is to give SVM novices a recipe for rapidly obtaining
acceptable results.
Although users do not need to understand the underlying theory behind SVM, we
briefly introduce the basics necessary for explaining our procedure. A classification
task usually involves separating data into training and testing sets. Each instance
in the training set contains one target value (i.e. the class labels) and several
attributes (i.e. the features or observed variables). The goal of SVM is to produce
a model (based on the training data) which predicts the target values of the test data
given only the test data attributes.
Given a training set of instance-label pairs (xi , yi ), i = 1, . . . , l where xi Rn and
y {1, 1}l , the support vector machines (SVM) (Boser et al., 1992; Cortes and
Vapnik, 1995) require the solution of the following optimization problem:
l
min
w,b,
subject to
X
1 T
w w+C
i
2
i=1
yi (wT (xi ) + b) 1 i ,
i 0.
1
(1)
Astroparticle1
Bioinformatics2
Vehicle3
4,000
04
41
4
20
21
2
3
2
Accuracy
by our
procedure
75.2%
96.9%
36%
85.2%
4.88%
87.8%
Here training vectors xi are mapped into a higher (maybe infinite) dimensional space
by the function . SVM finds a linear separating hyperplane with the maximal margin
in this higher dimensional space. C > 0 is the penalty parameter of the error term.
Furthermore, K(xi , xj ) (xi )T (xj ) is called the kernel function. Though new
kernels are being proposed by researchers, beginners may find in SVM books the
following four basic kernels:
linear: K(xi , xj ) = xTi xj .
polynomial: K(xi , xj ) = (xi T xj + r)d , > 0.
radial basis function (RBF): K(xi , xj ) = exp(kxi xj k2 ), > 0.
sigmoid: K(xi , xj ) = tanh(xi T xj + r).
Here, , r, and d are kernel parameters.
1.1
Real-World Examples
Table 1 presents some real-world examples. These data sets are supplied by our users
who could not obtain reasonable accuracy in the beginning. Using the procedure
illustrated in this guide, we help them to achieve better performance. Details are in
Appendix A.
These data sets are at http://www.csie.ntu.edu.tw/~cjlin/papers/guide/
data/
1
1.2
Proposed Procedure
Data Preprocessing
2.1
Categorical Feature
SVM requires that each data instance is represented as a vector of real numbers.
Hence, if there are categorical attributes, we first have to convert them into numeric
data. We recommend using m numbers to represent an m-category attribute. Only
one of the m numbers is one, and others are zero. For example, a three-category
attribute such as {red, green, blue} can be represented as (0,0,1), (0,1,0), and (1,0,0).
Our experience indicates that if the number of values in an attribute is not too large,
this coding might be more stable than using a single number.
5
The best parameter might be affected by the size of data set but in practice the one obtained
from cross-validation is already suitable for the whole training set.
2.2
Scaling
Scaling before applying SVM is very important. Part 2 of Sarles Neural Networks
FAQ Sarle (1997) explains the importance of this and most of considerations also apply to SVM. The main advantage of scaling is to avoid attributes in greater numeric
ranges dominating those in smaller numeric ranges. Another advantage is to avoid
numerical difficulties during the calculation. Because kernel values usually depend on
the inner products of feature vectors, e.g. the linear kernel and the polynomial kernel, large attribute values might cause numerical problems. We recommend linearly
scaling each attribute to the range [1, +1] or [0, 1].
Of course we have to use the same method to scale both training and testing
data. For example, suppose that we scaled the first attribute of training data from
[10, +10] to [1, +1]. If the first attribute of testing data lies in the range [11, +8],
we must scale the testing data to [1.1, +0.8]. See Appendix B for some real examples.
Model Selection
Though there are only four common kernels mentioned in Section 1, we must decide
which one to try first. Then the penalty parameter C and kernel parameters are
chosen.
3.1
RBF Kernel
In general, the RBF kernel is a reasonable first choice. This kernel nonlinearly maps
samples into a higher dimensional space so it, unlike the linear kernel, can handle the
case when the relation between class labels and attributes is nonlinear. Furthermore,
the linear kernel is a special case of RBF Keerthi and Lin (2003) since the linear
kernel with a penalty parameter C has the same performance as the RBF kernel with
some parameters (C, ). In addition, the sigmoid kernel behaves like RBF for certain
parameters (Lin and Lin, 2003).
The second reason is the number of hyperparameters which influences the complexity of model selection. The polynomial kernel has more hyperparameters than
the RBF kernel.
Finally, the RBF kernel has fewer numerical difficulties. One key point is 0 <
Kij 1 in contrast to polynomial kernels of which kernel values may go to infinity
(xi T xj + r > 1) or zero (xi T xj + r < 1) while the degree is large. Moreover, we
must note that the sigmoid kernel is not valid (i.e. not the inner product of two
3.2
There are two parameters for an RBF kernel: C and . It is not known beforehand
which C and are best for a given problem; consequently some kind of model selection
(parameter search) must be done. The goal is to identify good (C, ) so that the
classifier can accurately predict unknown data (i.e. testing data). Note that it may
not be useful to achieve high training accuracy (i.e. a classifier which accurately
predicts training data whose class labels are indeed known). As discussed above, a
common strategy is to separate the data set into two parts, of which one is considered
unknown. The prediction accuracy obtained from the unknown set more precisely
reflects the performance on classifying an independent data set. An improved version
of this procedure is known as cross-validation.
In v-fold cross-validation, we first divide the training set into v subsets of equal
size. Sequentially one subset is tested using the classifier trained on the remaining
v 1 subsets. Thus, each instance of the whole training set is predicted once so the
cross-validation accuracy is the percentage of data which are correctly classified.
The cross-validation procedure can prevent the overfitting problem. Figure 1
represents a binary classification problem to illustrate this issue. Filled circles and
triangles are the training data while hollow circles and triangles are the testing data.
The testing accuracy of the classifier in Figures 1a and 1b is not good since it overfits
the training data. If we think of the training and testing data in Figure 1a and 1b
as the training and validation sets in cross-validation, the accuracy is not good. On
the other hand, the classifier in 1c and 1d does not overfit the training data and gives
better cross-validation as well as testing accuracy.
We recommend a grid-search on C and using cross-validation. Various pairs
of (C, ) values are tried and the one with the best cross-validation accuracy is
picked. We found that trying exponentially growing sequences of C and is a
practical method to identify good parameters (for example, C = 25 , 23 , . . . , 215 ,
= 215 , 213 , . . . , 23 ).
The grid-search is straightforward but seems naive. In fact, there are several
advanced methods which can save computational cost by, for example, approximating
the cross-validation rate. However, there are two motivations why we prefer the simple
5
using a coarse grid first. After identifying a better region on the grid, a finer grid
search on that region can be conducted. To illustrate this, we do an experiment on
the problem german from the Statlog collection (Michie et al., 1994). After scaling
this set, we first use a coarse grid (Figure 2) and find that the best (C, ) is (23 , 25 )
with the cross-validation rate 77.5%. Next we conduct a finer grid search on the
neighborhood of (23 , 25 ) (Figure 3) and obtain a better cross-validation rate 77.6%
at (23.25 , 25.25 ). After the best (C, ) is found, the whole training set is trained again
to generate the final classifier.
The above approach works well for problems with thousands or more data points.
For very large data sets a feasible approach is to randomly choose a subset of the
data set, conduct grid-search on them, and then do a better-region-only grid-search
on the complete data set.
Discussion
In some situations the above proposed procedure is not good enough, so other techniques such as feature selection may be needed. These issues are beyond the scope of
this guide. Our experience indicates that the procedure works well for data which do
not have many features. If there are thousands of attributes, there may be a need to
choose a subset of them before giving the data to SVM.
Acknowledgments
We thank all users of our SVM software LIBSVM and BSVM, who helped us to
identify possible difficulties encountered by beginners. We also thank some users (in
particular, Robert Campbell) for proofreading the paper.
In this appendix we compare accuracy by the proposed procedure with that often
used by general beginners. Experiments are on the three problems mentioned in
Table 1 by using the software LIBSVM (Chang and Lin, 2001). For each problem, we
first list the accuracy by direct training and testing. Secondly, we show the difference
in accuracy with and without scaling. From what has been discussed in Section 2.2,
the range of training set attributes must be saved so that we are able to restore
them while scaling the testing set. Thirdly, the accuracy by the proposed procedure
(scaling and then model selection) is presented. Finally, we demonstrate the use
of a tool in LIBSVM which does the whole procedure automatically. Note that a
similar parameter selection tool like the grid.py presented below is available in the
R-LIBSVM interface (see the function tune).
A.1
Astroparticle Physics
A.2
Bioinformatics
10
A.3
Vehicle
Section 2.2 stresses the importance of using the same scaling factors for training and
testing sets. We give a real example on classifying traffic light signals (courtesy of an
anonymous user) It is available at LIBSVM Data Sets.
If training and testing sets are separately scaled to [0, 1], the resulting accuracy is
lower than 70%.
$ ../svm-scale -l 0 svmguide4 > svmguide4.scale
$ ../svm-scale -l 0 svmguide4.t > svmguide4.t.scale
$ python easy.py svmguide4.scale svmguide4.t.scale
Accuracy = 69.2308% (216/312) (classification)
Using the same scaling factors for training and testing sets, we obtain much better
accuracy.
$ ../svm-scale -l 0 -s range4 svmguide4 > svmguide4.scale
$ ../svm-scale -r range4 svmguide4.t > svmguide4.t.scale
$ python easy.py svmguide4.scale svmguide4.t.scale
Accuracy = 89.4231% (279/312) (classification)
With the correct setting, the 10 features in svmguide4.t.scale have the following
maximal values:
0.7402, 0.4421, 0.6291, 0.8583, 0.5385, 0.7407, 0.3982, 1.0000, 0.8218, 0.9874
Clearly, the earlier way to scale the testing set to [0, 1] generates an erroneous set.
If the number of features is large, one may not need to map data to a higher dimensional space. That is, the nonlinear mapping does not improve the performance.
Using the linear kernel is good enough, and one only searches for the parameter C.
While Section 3.1 describes that RBF is at least as good as linear, the statement is
true only after searching the (C, ) space.
Next, we split our discussion to three parts:
12
C.1
Many microarray data in bioinformatics are of this type. We consider the Leukemia
data from the LIBSVM data sets (http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/
datasets). The training and testing sets have 38 and 34 instances, respectively. The
number of features is 7,129, much larger than the number of instances. We merge the
two files and compare the cross validation accuracy of using the RBF and the linear
kernels:
RBF kernel with parameter selection
$ cat leu leu.t > leu.combined
$ python grid.py leu.combined
C.2
Such data often occur in document classification. LIBSVM is not particularly good for
this type of problems. Fortunately, we have another software LIBLINEAR (Fan et al.,
2008), which is very suitable for such data. We illustrate the difference between
13
LIBSVM and LIBLINEAR using a document problem rcv1 train.binary from the
LIBSVM data sets. The numbers of instances and features are 20,242 and 47,236,
respectively.
$ time libsvm-2.85/svm-train -c 4 -t 0 -e 0.1 -m 800 -v 5 rcv1_train.binary
Cross Validation Accuracy = 96.8136%
345.569s
$ time liblinear-1.21/train -c 4 -e 0.1 -v 5 rcv1_train.binary
Cross Validation Accuracy = 97.0161%
2.944s
For five-fold cross validation, LIBSVM takes around 350 seconds, but LIBLINEAR uses
only 3. Moreover, LIBSVM consumes more memory as we allocate some spaces to
store recently used kernel elements (see -m 800). Clearly, LIBLINEAR is much faster
than LIBSVM to obtain a model with comparable accuracy.
LIBLINEAR is efficient for large-scale document classification. Let us consider a
large set rcv1 test.binary with 677,399 instances.
$ time liblinear-1.21/train -c 0.25 -v 5 rcv1_test.binary
Cross Validation Accuracy = 97.8538%
68.84s
Note that reading the data takes most of the time. The training of each training/validation split is less than four seconds.
C.3
As the number of features is small, one often maps data to higher dimensional spaces
(i.e., using nonlinear kernels). However, if you really would like to use the linear
kernel, you may use LIBLINEAR with the option -s 2. When the number of features
is small, it is often faster than the default -s 1. Consider the data http://www.csie.
ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/covtype.libsvm.binary.scale.
bz2. The number of instances 581,012 is much larger than the number of features 54.
We run LIBLINEAR with -s 1 (default) and -s 2.
$ time liblinear-1.21/train -c 4 -v 5 -s 2 covtype.libsvm.binary.scale
Cross Validation Accuracy = 75.67%
67.224s
$ time liblinear-1.21/train -c 4 -v 5 -s 1 covtype.libsvm.binary.scale
14
References
B. E. Boser, I. Guyon, and V. Vapnik. A training algorithm for optimal margin
classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning
Theory, pages 144152. ACM Press, 1992.
C.-C. Chang and C.-J. Lin. LIBSVM: a library for support vector machines, 2001.
Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
C. Cortes and V. Vapnik. Support-vector network. Machine Learning, 20:273297,
1995.
R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin. LIBLINEAR: A
library for large linear classification. Journal of Machine Learning Research, 9:1871
1874, 2008. URL http://www.csie.ntu.edu.tw/~cjlin/papers/liblinear.
pdf.
J. L. Gardy, C. Spencer, K. Wang, M. Ester, G. E. Tusnady, I. Simon, S. Hua,
K. deFays, C. Lambert, K. Nakai, and F. S. Brinkman. PSORT-B: improving
protein subcellular localization prediction for gram-negative bacteria. Nucleic Acids
Research, 31(13):36133617, 2003.
S. S. Keerthi and C.-J. Lin. Asymptotic behaviors of support vector machines with
Gaussian kernel. Neural Computation, 15(7):16671689, 2003.
H.-T. Lin and C.-J. Lin. A study on sigmoid kernels for SVM and the training of nonPSD kernels by SMO-type methods. Technical report, Department of Computer
Science, National Taiwan University, 2003. URL http://www.csie.ntu.edu.tw/
~cjlin/papers/tanh.pdf.
D. Michie, D. J. Spiegelhalter, C. C. Taylor, and J. Campbell, editors. Machine
learning, neural and statistical classification. Ellis Horwood, Upper Saddle River,
NJ, USA, 1994. ISBN 0-13-106360-X. Data available at http://archive.ics.
uci.edu/ml/machine-learning-databases/statlog/.
15
16
Introduction
Support Vector Machines (SVMs) are a popular machine learning method for classification, regression, and other learning tasks. Since the year 2000, we have been developing the package LIBSVM as a library for support vector machines. The Web address
of the package is at http://www.csie.ntu.edu.tw/~cjlin/libsvm. LIBSVM is currently one of the most widely used SVM software. In this article,1 we present all
implementation details of LIBSVM. However, this article does not intend to teach
the practical use of LIBSVM. For instructions of using LIBSVM, see the README file
included in the package, the LIBSVM FAQ,2 and the practical guide by Hsu et al.
(2003). An earlier version of this article was published in Chang and Lin (2011).
LIBSVM supports the following learning tasks.
1
This LIBSVM implementation document was created in 2001 and has been maintained at http:
//www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf.
2
LIBSVM FAQ: http://www.csie.ntu.edu.tw/~cjlin/libsvm/faq.html.
Table 1: Representative works in some domains that have successfully used LIBSVM.
Representative works
Domain
Computer vision
LIBPMK (Grauman and Darrell, 2005)
Natural language processing Maltparser (Nivre et al., 2007)
Neuroimaging
PyMVPA (Hanke et al., 2009)
Bioinformatics
BDVal (Dorff et al., 2010)
1. SVC: support vector classification (two-class and multi-class).
2. SVR: support vector regression.
3. One-class SVM.
A typical use of LIBSVM involves two steps: first, training a data set to obtain a
model and second, using the model to predict information of a testing data set. For
SVC and SVR, LIBSVM can also output probability estimates. Many extensions of
LIBSVM are available at libsvmtools.3
The LIBSVM package is structured as follows.
1. Main directory: core C/C++ programs and sample data. In particular, the file
svm.cpp implements training and testing algorithms, where details are described
in this article.
2. The tool sub-directory: this sub-directory includes tools for checking data
format and for selecting SVM parameters.
3. Other sub-directories contain pre-built binary files and interfaces to other languages/software.
LIBSVM has been widely used in many areas. From 2000 to 2010, there were
more than 250,000 downloads of the package. In this period, we answered more than
10,000 emails from users. Table 1 lists representative works in some domains that
have successfully used LIBSVM.
This article is organized as follows. In Section 2, we describe SVM formulations
supported in LIBSVM: C-support vector classification (C-SVC), -support vector
classification (-SVC), distribution estimation (one-class SVM), -support vector regression (-SVR), and -support vector regression (-SVR). Section 3 then discusses
performance measures, basic usage, and code organization. All SVM formulations
3
supported in LIBSVM are quadratic minimization problems. We discuss the optimization algorithm in Section 4. Section 5 describes two implementation techniques
to reduce the running time for minimizing SVM quadratic problems: shrinking and
caching. LIBSVM provides some special settings for unbalanced data; details are
in Section 6. Section 7 discusses our implementation for multi-class classification.
Section 8 presents how to transform SVM decision values into probability values. Parameter selection is important for obtaining good SVM models. Section 9 presents a
simple and useful parameter selection tool in LIBSVM. Finally, Section 10 concludes
this work.
SVM Formulations
LIBSVM supports various SVM formulations for classification, regression, and distribution estimation. In this section, we present these formulations and give corresponding references. We also show performance measures used in LIBSVM.
2.1
X
1 T
w w+C
i
2
i=1
min
w,b,
subject to
(1)
yi (wT (xi ) + b) 1 i ,
i 0, i = 1, . . . , l,
where (xi ) maps xi into a higher-dimensional space and C > 0 is the regularization
parameter. Due to the possible high dimensionality of the vector variable w, usually
we solve the following dual problem.
min
subject to
1 T
Q eT
2
y T = 0,
0 i C,
(2)
i = 1, . . . , l,
After problem (2) is solved, using the primal-dual relationship, the optimal w
satisfies
w=
l
X
yi i (xi )
(3)
i=1
!
yi i K(xi , x) + b .
i=1
2.2
The -support vector classification (Scholkopf et al., 2000) introduces a new parameter
(0, 1]. It is proved that an upper bound on the fraction of training errors and
a lower bound of the fraction of support vectors.
Given training vectors xi Rn , i = 1, . . . , l, in two classes, and a vector y Rl
such that yi {1, 1}, the primal optimization problem is
l
min
w,b,,
subject to
1 T
1X
w w +
i
2
l i=1
yi (wT (xi ) + b) i ,
i 0, i = 1, . . . , l,
(4)
0.
subject to
1 T
Q
2
0 i 1/l,
eT ,
i = 1, . . . , l,
(5)
y T = 0,
where Qij = yi yj K(xi , xj ). Chang and Lin (2001) show that problem (5) is feasible
if and only if
2 min(#yi = +1, #yi = 1)
1,
l
so the usable range of is smaller than (0, 1].
4
In LIBSVM, any integer can be a label name, so we map label names to 1 by assigning the first
training instance to have y1 = +1.
l
X
!
yi i K(xi , x) + b .
i=1
subject to
1 T
Q
2
0
i 1,
= l,
eT
i = 1, . . . , l,
(6)
= 0.
yT
If is optimal for the dual problem (5) and is optimal for the primal problem
(4), Chang and Lin (2001) show that / is an optimal solution of C-SVM with
C = 1/(l). Thus, in LIBSVM, we output (/, b/) in the model.5
2.3
One-class SVM was proposed by Scholkopf et al. (2001) for estimating the support of
a high-dimensional distribution. Given training vectors xi Rn , i = 1, . . . , l without
any class information, the primal problem of one-class SVM is
l
1 X
1 T
w w+
i
2
l i=1
min
w,,
wT (xi ) i ,
subject to
i 0, i = 1, . . . , l.
The dual problem is
min
subject to
1 T
Q
2
0 i 1/(l), i = 1, . . . , l,
(7)
eT = 1,
where Qij = K(xi , xj ) = (xi )T (xj ). The decision function is
!
l
X
sgn
i K(xi , x) .
i=1
5
= l, we have / = /
. Hence, in
More precisely, solving (6) obtains = l. Because
.
LIBSVM, we calculate /
subject to
1 T
Q
2
0 i 1,
i = 1, . . . , l,
(8)
eT = l.
2.4
min
w,b,,
subject to
X
X
1 T
w w+C
i + C
i
2
i=1
i=1
wT (xi ) + b zi + i ,
zi wT (xi ) b + i ,
i , i 0, i = 1, . . . , l.
min
subject to
X
X
1
( )T Q( ) +
(i + i ) +
zi (i i )
2
i=1
i=1
eT ( ) = 0,
(9)
0 i , i C, i = 1, . . . , l,
where Qij = K(xi , xj ) (xi )T (xj ).
After solving problem (9), the approximate function is
l
X
(i + i )K(xi , x) + b.
i=1
2.5
Similar to -SVC, for regression, Scholkopf et al. (2000) use a parameter (0, 1]
to control the number of support vectors. The parameter in -SVR becomes a
1 T
1X
w w + C( +
(i + i ))
2
l i=1
min
w,b,, ,
(wT (xi ) + b) zi + i ,
subject to
zi (wT (xi ) + b) + i ,
i , i 0, i = 1, . . . , l,
0.
subject to
1
( )T Q( ) + z T ( )
2
eT ( ) = 0, eT ( + ) C,
0 i , i C/l,
(10)
i = 1, . . . , l.
Similar to -SVC, Chang and Lin (2002) show that the inequality eT (+ ) C
can be replaced by an equality. Moreover, C/l may be too small because users often
choose C to be a small constant like one. Thus, in LIBSVM, we treat the userspecified regularization parameter as C/l. That is, C = C/l is what users specified
and LIBSVM solves the following problem.
min
subject to
1
( )T Q( ) + z T ( )
2
eT ( ) = 0, eT ( + ) = Cl,
0 i , i C,
i = 1, . . . , l.
This section describes LIBSVMs evaluation measures, shows some simple examples
of running LIBSVM, and presents the code structure.
7
3.1
Performance Measures
After solving optimization problems listed in previous sections, users can apply decision functions to predict labels (target values) of testing data. Let x1 , . . . , xl be
the testing data and f (x1 ), . . . , f (xl ) be decision values (target values for regression)
predicted by LIBSVM. If the true labels (true target values) of testing data are known
and denoted as yi , . . . , yl , we evaluate the prediction results by the following measures.
3.1.1
Classification
Accuracy
# correctly predicted data
=
100%
# total testing data
3.1.2
Regression
LIBSVM outputs MSE (mean squared error) and r2 (squared correlation coefficient).
1X
MSE =
(f (xi ) yi )2 ,
l i=1
2
P
l l f (xi )yi Pl f (xi ) Pl yi
i=1
i=1
i=1
P
r2 = P
2 .
Pl
P
2
l
l
2
l l f (xi )2
y
l
y
f
(x
)
i
i=1 i
i=1 i
i=1
i=1
3.2
While detailed instructions of using LIBSVM are available in the README file of the
package and the practical guide by Hsu et al. (2003), here we give a simple example.
LIBSVM includes a sample data set heart scale of 270 instances. We split the
data to a training set heart scale.tr (170 instances) and a testing set heart scale.te.
$ python tools/subset.py heart_scale 170 heart_scale.tr heart_scale.te
The command svm-train solves an SVM optimization problem to produce a model.6
$ ./svm-train heart_scale.tr
*
optimization finished, #iter = 87
nu = 0.471645
6
The default solver is C-SVC using the RBF kernel (48) with C = 1 and = 1/n.
3.3
Code Organization
All LIBSVMs training and testing algorithms are implemented in the file svm.cpp.
The two main sub-routines are svm train and svm predict. The training procedure
is more sophisticated, so we give the code organization in Figure 1.
From Figure 1, for classification, svm train decouples a multi-class problem to
two-class problems (see Section 7) and calls svm train one several times. For regression and one-class SVM, it directly calls svm train one. The probability outputs
for classification and regression are also handled in svm train. Then, according
to the SVM formulation, svm train one calls a corresponding sub-routine such as
solve c svc for C-SVC and solve nu svc for -SVC. All solve * sub-routines call
the solver Solve after preparing suitable input values. The sub-routine Solve minimizes a general form of SVM optimization problems; see (11) and (22). Details of
the sub-routine Solve are described in Sections 4-6.
This section discusses algorithms used in LIBSVM to solve dual quadratic problems
listed in Section 2. We split the discussion to two parts. The first part considers
optimization problems with one linear constraint, while the second part checks those
with two linear constraints.
svm train
Solve
Figure 1: LIBSVMs code organization for training. All sub-routines are in svm.cpp.
4.1
We consider the following general form of C-SVC, -SVR, and one-class SVM.
min
subject to
f ()
y T = ,
(11)
0 t C, t = 1, . . . , l,
where
1
f () T Q + pT
2
and yt = 1, t = 1, . . . , l. The constraint y T = 0 is called a linear constraint. It can
be clearly seen that C-SVC and one-class SVM are already in the form of problem
(11). For -SVR, we use the following reformulation of Eq. (9).
T
1 T T Q Q
T
T
T
( ) ,
+ e z , e + z
min
Q Q
,
2
subject to
yT
= 0,
0 t , t C, t = 1, . . . , l,
where
y = [1, . . . , 1, 1, . . . , 1]T .
| {z } | {z }
l
10
4.1.1
The main difficulty for solving problem (11) is that Q is a dense matrix and may be
too large to be stored. In LIBSVM, we consider a decomposition method to conquer
this difficulty. Some earlier works on decomposition methods for SVM include, for
example, Osuna et al. (1997a); Joachims (1998); Platt (1998); Keerthi et al. (2001);
Hsu and Lin (2002b). Subsequent developments include, for example, Fan et al.
(2005); Palagi and Sciandrone (2005); Glasmachers and Igel (2006). A decomposition
method modifies only a subset of per iteration, so only some columns of Q are
needed. This subset of variables, denoted as the working set B, leads to a smaller
optimization sub-problem. An extreme case of the decomposition methods is the
Sequential Minimal Optimization (SMO) (Platt, 1998), which restricts B to have only
two elements. Then, at each iteration, we solve a simple two-variable problem without
needing any optimization software. LIBSVM considers an SMO-type decomposition
method proposed in Fan et al. (2005).
Algorithm 1 (An SMO-type decomposition method in Fan et al., 2005)
1. Find 1 as the initial feasible solution. Set k = 1.
2. If k is a stationary point of problem (2), stop. Otherwise, find a two-element
working set B = {i, j} by WSS 1 (described in Section 4.1.2). Define N
{1, . . . , l}\B. Let kB and kN be sub-vectors of k corresponding to B and N ,
respectively.
3. If aij Kii + Kjj 2Kij > 0,7
Solve the following sub-problem with the variable B = [i j ]T .
Qii Qij i
1
k T i
i j
min
+ (pB + QBN N )
Qij Qjj j
j
i ,j
2
subject to
0 i , j C,
(12)
yi i + yj j = y TN kN ,
else
7
11
B . If Q is PSD, then aij > 0. Thus sub-problem (13) is used only to handle the
situation where Q is non-PSD.
4.1.2
(14)
iIlow ()
and
Iup () {t | t < C, yt = 1 or t > 0, yt = 1}, and
Ilow () {t | t < C, yt = 1 or t > 0, yt = 1}.
12
(16)
(17)
a
ts
ats
(18)
if ats > 0,
otherwise.
Select
i arg max{yt t f (k ) | t Iup (k )},
t
2
bit
k
k
k
j arg min
| t Ilow ( ), yt t f ( ) < yi i f ( ) .
t
a
it
(19)
Details of solving the two-variable sub-problem in Eqs. (12) and (13) are deferred to
Section 6, where a more general sub-problem is discussed.
4.1.4
From the discussion in Sections 4.1.1 and 4.1.2, the main operations per iteration
are on finding QBN kN + pB for constructing the sub-problem (12), and calculating
13
f (k ) for the working set selection and the stopping condition. These two operations can be considered together because
QBN kN + pB = B f (k ) QBB kB
(20)
(21)
and
where |B| |N | and Q:,B is the sub-matrix of Q including columns in B. If at the
kth iteration we already have f (k ), then Eq. (20) can be used to construct the
sub-problem. After the sub-problem is solved, Eq. (21) is employed to have the next
f (k+1 ). Therefore, LIBSVM maintains the gradient throughout the decomposition
method.
4.1.5
The Calculation of b or
After the solution of the dual optimization problem is obtained, the variables b or
must be calculated as they are used in the decision function.
Note that b of C-SVC and -SVR plays the same role as in one-class SVM, so
we define = b and discuss how to find . If there exists i such that 0 < i < C,
then from the KKT condition (16), = yi i f (). In LIBSVM, for numerical stability,
we average all these values.
P
=
yi i f ()
.
|{i | 0 < i < C}|
i:0<i <C
For the situation that no i satisfying 0 < i < C, the KKT condition (16) becomes
M () = max{yi i f () | i = 0, yi = 1 or i = C, yi = 1}
m() = min{yi i f () | i = 0, yi = 1 or i = C, yi = 1}.
We take the midpoint of the preceding range.
4.1.6
Initial Values
Algorithm 1 requires an initial feasible . For C-SVC and -SVR, because the zero
vector is feasible, we select it as the initial .
For one-class SVM, the scaled form (8) requires that
0 i 1,
and
l
X
i=1
14
i = l.
We let the first blc elements have i = 1 and the (blc + 1)st element have i =
l blc.
4.1.7
Fan et al. (2005, Section III) and Chen et al. (2006) discuss the convergence of Algorithm 1 in detail. For the rate of linear convergence, List and Simon (2009) prove a
result without making the assumption used in Chen et al. (2006).
4.2
From problems (6) and (10), both -SVC and -SVR can be written as the following
general form.
min
subject to
1 T
Q + pT
2
y T = 1 ,
(22)
eT = 2 ,
0 t C, t = 1, . . . , l.
The main difference between problems (11) and (22) is that (22) has two linear
constraints y T = 1 and eT = 2 . The optimization algorithm is very similar to
that for (11), so we describe only differences.
4.2.1
Let f () be the objective function of problem (22). By the same derivation in Section 4.1.2, The KKT condition of problem (22) implies that there exist b and such
that
(
0
i f () + byi
0
if i < C,
if i > 0.
(23)
Define
r1 b and r2 + b.
(24)
If yi = 1, (23) becomes
(
0
i f () r1
0
15
if i < C,
if i > 0.
(25)
if yi = 1, (23) becomes
(
0
i f () r2
0
if i < C,
if i > 0.
(26)
(27)
where
mp ()
mn ()
max
iIup (),yi =1
yi i f (),
max
iIup (),yi =1
yi i f (),
Mp ()
Mn ()
min
iIlow (),yi =1
yi i f (), and
min
iIlow (),yi =1
yi i f ().
b2ip t
a
ip t
2. Find
in arg mn (k ),
2
bi n t
k
k
k
jn arg min
| yt = 1, t Ilow ( ), yt t f ( ) < yin in f ( ) .
t
a
in t
aij .
3. Return {ip , jp } or {in , jn } depending on which one gives smaller b2ij /
4.2.2
We have shown that the KKT condition of problem (22) implies Eqs. (25) and (26)
according to yi = 1 and 1, respectively. Now we consider the case of yi = 1. If
there exists i such that 0 < i < C, then we obtain r1 = i f (). In LIBSVM, for
numerical stability, we average these values.
P
r1 =
i f ()
.
|{i | 0 < i < C, yi = 1}|
i:0<i <C,yi =1
i =C,yi =1
i f () r1
16
min
i =0,yi =1
i f ().
r1 + r2
r1 r2
and b =
.
2
2
Initial Values
i =
i:yi =1
l
.
2
We let the first l/2 elements of i with yi = 1 to have the value one.8 The situation
for yi = 1 is similar. The same setting is applied to -SVR.
This section discusses two implementation tricks (shrinking and caching) for the decomposition method and investigates the computational complexity of Algorithm 1.
5.1
Shrinking
An optimal solution of the SVM dual problem may contain some bounded elements
(i.e., i = 0 or C). These elements may have already been bounded in the middle of
the decomposition iterations. To save the training time, the shrinking technique tries
to identify and remove some bounded elements, so a smaller optimization problem is
solved (Joachims, 1998). The following theorem theoretically supports the shrinking
technique by showing that at the final iterations of Algorithm 1 in Section 4.1.2, only
a small set of variables is still changed.
Theorem 5.1 (Theorem IV in Fan et al., 2005) Consider problem (11) and assume Q is positive semi-definite.
I {i | yi i f ()
Further, for every i I, problem (11) has a unique and bounded optimal solution
at i .
8
Special care must be made as l/2 may not be an integer. See also Section 4.1.6.
17
k k:
ik
i 6 {t | M (k ) yt t f (k ) m(k )}.
If we denote A as the set containing elements not shrunk at the kth iteration,
then instead of solving problem (11), the decomposition method works on a smaller
problem.
min
A
subject to
1 T
QAA A + (pA + QAN kN )T A
2 A
0 i C, i A,
(28)
y TA A = y TN kN ,
where N = {1, . . . , l}\A is the set of shrunk variables. Note that in LIBSVM, we
always rearrange elements of , y, and p to maintain that A = {1, . . . , |A|}. Details
of the index rearrangement are in Section 5.4.
After solving problem (28), we may find that some elements are wrongly shrunk.
When that happens, the original problem (11) is reoptimized from a starting point
A
= [
N ], where A is optimal for problem (28) and N corresponds to shrunk
bounded variables.
In LIBSVM, we start the shrinking procedure in an early stage. The procedure is
as follows.
1. After every min(l, 1000) iterations, we try to shrink some variables. Note that
throughout the iterative process, we have
m(k ) > M (k )
(29)
because the condition (17) is not satisfied yet. Following Theorem 5.1, we
conjecture that variables in the following set can be shrunk.
{t | yt t f (k ) > m(k ), t Ilow (k ), tk is bounded}
{t | yt t f (k ) < M (k ), t Iup (k ), tk is bounded}
= {t | yt t f (k ) > m(k ), tk = C, yt = 1 or tk = 0, yt = 1}
(30)
{t | yt t f (k ) < M (k ), tk = 0, yt = 1 or tk = C, yt = 1}.
Thus, the size of the set A is gradually reduced in every min(l, 1000) iterations.
The problem (28), and the way of calculating m(k ) and M (k ) are adjusted
accordingly.
18
2. The preceding shrinking strategy is sometimes too aggressive. Hence, when the
decomposition method achieves the following condition for the first time.
m(k ) M (k ) + 10,
(31)
(32)
of the smaller problem (28) is reached, we must check if the stopping condition
of the original problem (11) has been satisfied. If not, then we reactivate all
variables by setting A = {1, . . . , l} and start the same shrinking procedure on
the problem (28).
Note that in solving the shrunk problem (28), we only maintain its gradient
QAA A + QAN N + pA (see also Section 4.1.4). Hence, when we reactivate
all variables to reoptimize the problem (11), we must reconstruct the whole
gradient f (). Details are discussed in Section 5.3.
For -SVC and -SVR, because the stopping condition (27) is different from (17),
variables being shrunk are different from those in (30). For yt = 1, we shrink elements
in the following set
{t | yt t f (k ) > mp (k ), t = C, yt = 1}
{t | yt t f (k ) < Mp (k ), t = 0, yt = 1}.
For yt = 1, we consider the following set.
{t | yt t f (k ) > mn (k ), t = 0, yt = 1}
{t | yt t f (k ) < Mn (k ), t = C, yt = 1}.
5.2
Caching
Caching is an effective technique for reducing the computational time of the decomposition method. Because Q may be too large to be stored in the computer memory, Qij
elements are calculated as needed. We can use available memory (called kernel cache)
to store some recently used Qij (Joachims, 1998). Then, some kernel elements may
19
not need to be recalculated. Theorem 5.1 also supports the use of caching because
in final iterations, only certain columns of the matrix Q are still needed. If the cache
already contains these columns, we can save kernel evaluations in final iterations.
In LIBSVM, we consider a simple least-recent-use caching strategy. We use a
circular list of structures, where each structure is defined as follows.
struct head_t
{
head_t *prev, *next;
// a circular list
Qfloat *data;
int len;
};
A structure stores the first len elements of a kernel column. Using pointers prev and
next, it is easy to insert or delete a column. The circular list is maintained so that
structures are ordered from the least-recent-used one to the most-recent-used one.
Because of shrinking, columns cached in the computer memory may be in different
length. Assume the ith column is needed and Q1:t,i have been cached. If t |A|, we
calculate Qt+1:|A|,i and store Q1:|A|,i in the cache. If t > |A|, the desired Q1:|A|,i are
already in the cache. In this situation, we do not change the cached contents of the
ith column.
5.3
Qij , i = 1, . . . , l.
(33)
j:j =C
Then, for i
/ A,
i f () =
l
X
i + pi +
Qij j + pi = G
j=1
X
j:jA
0<j <C
Qij j .
(34)
The calculation of f () via Eq. (34) involves a two-level loop over i and j.
Using i or j first may result in a very different number of Qij evaluations. We discuss
the differences next.
1. i first: for |A| + 1 i l, calculate Qi,1:|A| . Although from Eq. (34), only
{Qij | 0 < j < C, j A} are needed, our implementation obtains all Qi,1:|A|
(i.e., {Qij | j A}). Hence, this case needs at most
(l |A|) |A|
(35)
kernel evaluations. Note that LIBSVM uses a column-based caching implementation. Due to the symmetry of Q, Qi,1:|A| is part of Qs ith column and may
have been cached. Thus, Eq. (35) is only an upper bound.
2. j first: let
F {j | 1 j |A| and 0 < j < C}.
For each j F , calculate Q1:l,j . Though only Q|A|+1:l,j is needed in calculating
i f (), i = |A| + 1, . . . , l, we must get the whole column because of our cache
implementation.9 Thus, this strategy needs no more than
l |F |
(36)
kernel evaluations. This is an upper bound because certain kernel columns (e.g.,
Q1:|A|,j , j A) may be already in the cache and do not need to be recalculated.
We may choose a method by comparing (35) and (36). However, the decision depends
on whether Qs elements have been cached. If the cache is large enough, then elements
of Qs first |A| columns tend to be in the cache because they have been used recently.
In contrast, Qi,1:|A| , i
/ A needed by method 1 may be less likely in the cache because
columns not in A are not used to solve problem (28). In such a situation, method 1
may require almost (l |A|) |A| kernel valuations, while method 2 needs much fewer
evaluations than l |F |.
Because method 2 takes an advantage of the cache implementation, we slightly
lower the estimate in Eq. (36) and use the following rule to decide the method of
calculating Eq. (34):
If (l/2) |F | > (l |A|) |A|
use method 1
Else
use method 2
9
21
This rule may not give the optimal choice because we do not take the cache contents
into account. However, we argue that in the worst scenario, the selected method by
the preceding rule is only slightly slower than the other method. This result can be
proved by making the following assumptions.
A LIBSVM training procedure involves only two gradient reconstructions. The
first is performed when the 10 tolerance is achieved; see Eq. (31). The second
is in the end of the training procedure.
Our rule assigns the same method to perform the two gradient reconstructions.
Moreover, these two reconstructions cost a similar amount of time.
We refer to total training time of method x as the whole LIBSVM training time
(where method x is used for reconstructing gradients), and reconstruction time of
method x as the time of one single gradient reconstruction via method x. We then
consider two situations.
1. Method 1 is chosen, but method 2 is better.
We have
Total time of method 1
(Total time of method 2) + 2 (Reconstruction time of method 1)
2 (Total time of method 2).
(37)
(38)
(39)
Table 2: A comparison between two gradient reconstruction methods. The decomposition method reconstructs the gradient twice after satisfying conditions (31) and
(32). We show in each row the number of kernel evaluations of a reconstruction. We
check two cache sizes to reflect the situations with/without enough cache. The last
two rows give the total training time (gradient reconstructions and other operations)
in seconds. We use the RBF kernel K(xi , xj ) = exp(kxi xj k2 ).
(a) a7a: C = 1, = 4, = 0.001.
Cache = 1,000 MB
Cache = 10 MB
l = 16, 100
Method 2
Reconstruction |F |
|A| Method 1 Method 2 Method 1
First
10,597 12,476
0 21,470,526 45,213,024 170,574,272
Second
10,630 12,476
0
0 45,213,024 171,118,048
102s
108s
341s
422s
Training time
No shrinking: 111s
No shrinking: 381s
(b) ijcnn1: C = 16, = 4, = 0.5.
l = 49, 900
Cache = 1,000 MB
Cache = 10 MB
Method 1 Method 2
Method 1
Method 2
Reconstruction |F |
|A|
First
1,767 43,678 274,297,840 5,403,072 275,695,536 88,332,330
Second
2,308 6,023 263,843,538 28,274,195 264,813,241 115,346,805
189s
46s
203s
116s
Training time
No shrinking: 42s
No shrinking: 87s
columns of Q are not in the cache either and (l |A|) |A| kernel evaluations
are needed for method 1. Because l |F | 2 (l |A|) |A|,
(Reconstruction time of method 2) 2 (Reconstruction time of method 1).
Therefore,
Total time of method 2
(Total time of method 1) + 2 (Reconstruction time of method 1)
2 (Total time of method 1).
Table 2 compares the number of kernel evaluations in reconstructing the gradient.
We consider problems a7a and ijcnn1.10 Clearly, the proposed rule selects the better
method for both problems. We implement this technique after version 2.88 of LIBSVM.
10
Available at http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets.
23
5.4
Index Rearrangement
In solving the smaller problem (28), we need only indices in A (e.g., i , yi , and xi ,
where i A). Thus, a naive implementation does not access array contents in a
continuous manner. Alternatively, we can maintain A = {1, . . . , |A|} by rearranging
array contents. This approach allows a continuous access of array contents, but
requires costs for the rearrangement. We decide to rearrange elements in arrays
because throughout the discussion in Sections 5.2-5.3, we assume that a cached ith
kernel column contains elements from the first to the tth (i.e., Q1:t,i ), where t l. If
we do not rearrange indices so that A = {1, . . . , |A|}, then the whole column Q1:l,i
must be cached because l may be an element in A.
We rearrange indices by sequentially swapping pairs of indices. If t1 is going to
be shrunk, we find an index t2 that should stay and then swap them. Swapping two
elements in a vector or y is easy, but swapping kernel elements in the cache is more
expensive. That is, we must swap (Qt1 ,i , Qt2 ,i ) for every cached kernel column i. To
make the number of swapping operations small, we use the following implementation.
Starting from the first and the last indices, we identify the smallest t1 that should
leave the largest t2 that should stay. Then, (t1 , t2 ) are swapped and we continue the
same procedure to identify the next pair.
5.5
5.6
We found that if the number of iterations is large, then shrinking can shorten the
training time. However, if we loosely solve the optimization problem (e.g., by using
a large stopping tolerance ), the code without using shrinking may be much faster.
In this situation, because of the small number of iterations, the time spent on all
decomposition iterations can be even less than one single gradient reconstruction.
Table 2 compares the total training time with/without shrinking. For a7a, we
use the default = 0.001. Under the parameters C = 1 and = 4, the number
of iterations is more than 30,000. Then shrinking is useful. However, for ijcnn1, we
deliberately use a loose tolerance = 0.5, so the number of iterations is only around
4,000. Because our shrinking strategy is quite aggressive, before the first gradient
reconstruction, only QA,A is in the cache. Then, we need many kernel evaluations for
reconstructing the gradient, so the implementation with shrinking is slower.
If enough iterations have been run, most elements in A correspond to free i
(0 < i < C); i.e., A F . In contrast, if the number of iterations is small (e.g.,
ijcnn1 in Table 2), many bounded elements have not been shrunk and |F | |A|.
Therefore, we can check the relation between |F | and |A| to conjecture if shrinking
11
25
5.7
Computational Complexity
While Section 4.1.7 has discussed the asymptotic convergence and the local convergence rate of the decomposition method, in this section, we investigate the computational complexity.
From Section 4, two places consume most operations at each iteration: finding the
working set B by WSS 1 and calculating Q:,B (k+1
kB ) in Eq. (21).12 Each place
B
requires O(l) operations. However, if Q:,B is not available in the cache and assume
each kernel evaluation costs O(n), the cost becomes O(ln) for calculating a column
of kernel elements. Therefore, the complexity of Algorithm 1 is
1. #Iterations O(l) if most columns of Q are cached throughout iterations.
2. #Iterations O(nl) if columns of Q are not cached and each kernel evaluation
costs O(n).
Several works have studied the number of iterations of decomposition methods; see,
for example, List and Simon (2007). However, algorithms studied in these works
are slightly different from LIBSVM, so there is no theoretical result yet on LIBSVMs
number of iterations. Empirically, it is known that the number of iterations may
be higher than linear to the number of training data. Thus, LIBSVM may take
considerable training time for huge data sets. Many techniques, for example, Fine
and Scheinberg (2001); Lee and Mangasarian (2001); Keerthi et al. (2006); Segata and
Blanzieri (2010), have been developed to obtain an approximate model, but these are
beyond the scope of our discussion. In LIBSVM, we provide a simple sub-sampling
tool, so users can quickly train a small subset.
For some classification problems, numbers of data in different classes are unbalanced.
Some researchers (e.g., Osuna et al., 1997b, Section 2.5; Vapnik, 1998, Chapter 10.9)
12
Note that because |B| = 2, once the sub-problem has been constructed, solving it takes only a
constant number of operations (see details in Section 6).
26
have proposed using different penalty parameters in the SVM formulation. For example, the C-SVM problem becomes
X
X
1 T
i
i + C
w w + C+
2
y =1
y =1
min
w,b,
yi (w (xi ) + b) 1 i ,
subject to
(40)
i 0, i = 1, . . . , l,
where C + and C are regularization parameters for positive and negative classes,
respectively. LIBSVM supports this setting, so users can choose weights for classes.
The dual problem of problem (40) is
1 T
Q eT
2
0 i C + , if yi = 1,
min
subject to
0 i C , if yi = 1,
y T = 0.
A more general setting is to assign each instance xi a regularization parameter Ci .
If C is replaced by Ci , i = 1, . . . , l in problem (11), most results discussed in earlier
sections can be extended without problems.13 The major change of Algorithm 1 is
on solving the sub-problem (12), which now becomes
Qii Qij i
1
i j
min
+ (Qi,N N + pi )i + (Qj,N N + pj )j
Qji Qjj j
i ,j
2
subject to
yi i + yj j = y TN kN ,
(41)
0 i Ci , 0 j Cj .
Let i = ik + di and j = jk + dj . The sub-problem (41) can be written as
Qii Qij di
di
1
k
k
di dj
+ i f ( ) j f ( )
min
Qij Qjj dj
dj
di ,dj
2
subject to
yi di + yj dj = 0,
ik di Ci ik , jk dj Cj jk .
Define aij and bij as in Eq. (18), and di yi di , dj yj dj . Using di = dj , the
objective function can be written as
1 2
a
ij dj + bij dj .
2
13
27
(42)
These two values may need to be modified because of bound constraints. We first
consider the case of yi 6= yj and re-write Eq. (42) as
inew = ik + (i f (k ) j f (k ))/
aij ,
aij .
jnew = jk + (i f (k ) j f (k ))/
In the following figure, a box is generated according to bound constraints. An infeasible (inew , jnew ) must be in one of the four regions outside the following box.
NA
i j = Ci Cj
region II
Cj
region IV
region I
Ci
i j = 0
region III
i
NA
Note that (inew , jnew ) does not appear in the NA regions because (ik , jk ) is in the
box and
inew jnew = ik jk .
If (inew , jnew ) is in region I, we set
ik+1 = Ci and jk+1 = Ci (ik jk ).
Of course, we must identify the region that (inew , jnew ) resides. For region I, we have
ik jk > Ci Cj and inew Ci .
Other cases are similar. We have the following pseudo code to identify which region
(inew , jnew ) is in and modify (inew , jnew ) to satisfy bound constraints.
if(y[i]!=y[j])
{
double quad_coef = Q_i[i]+Q_j[j]+2*Q_i[j];
if (quad_coef <= 0)
28
quad_coef = TAU;
double delta = (-G[i]-G[j])/quad_coef;
double diff = alpha[i] - alpha[j];
alpha[i] += delta;
alpha[j] += delta;
if(diff > 0)
{
if(alpha[j] < 0) // in region III
{
alpha[j] = 0;
alpha[i] = diff;
}
}
else
{
if(alpha[i] < 0) // in region IV
{
alpha[i] = 0;
alpha[j] = -diff;
}
}
if(diff > C_i - C_j)
{
if(alpha[i] > C_i) // in region I
{
alpha[i] = C_i;
alpha[j] = C_i - diff;
}
}
else
{
if(alpha[j] > C_j) // in region II
{
alpha[j] = C_j;
alpha[i] = C_j + diff;
}
}
}
If yi = yj , the derivation is the same.
Multi-class classification
LIBSVM implements the one-against-one approach (Knerr et al., 1990) for multiclass classification. Some early works of applying this strategy to SVM include, for
example, Kressel (1998). If k is the number of classes, then k(k 1)/2 classifiers are
constructed and each one trains data from two classes. For training data from the
29
ith and the jth classes, we solve the following two-class classification problem.
min
subject to
X
1 ij T ij
(w ) w + C
( ij )t
2
t
(wij )T (xt ) + bij 1 tij , if xt in the ith class,
(wij )T (xt ) + bij 1 + tij , if xt in the jth class,
tij 0.
Probability Estimates
SVM predicts only class label (target value for regression) without probability information. This section discusses the LIBSVM implementation for extending SVM to
give probability estimates. More details are in Wu et al. (2004) for classification and
in Lin and Weng (2004) for regression.
Given k classes of data, for any x, the goal is to estimate
pi = P (y = i | x),
i = 1, . . . , k.
Following the setting of the one-against-one (i.e., pairwise) approach for multi-class
classification, we first estimate pairwise class probabilities
rij P (y = i | y = i or j, x)
using an improved implementation (Lin et al., 2007) of Platt (2000). If f is the
decision value at x, then we assume
rij
1
,
1 + eAf+B
(43)
where A and B are estimated by minimizing the negative log likelihood of training
data (using their labels and decision values). It has been observed that decision values
30
from training may overfit the model (43), so we conduct five-fold cross-validation to
obtain decision values before minimizing the negative log likelihood. Note that if
some classes contain five or even fewer data points, the resulting model may not be
good. You can duplicate the data set so that each fold of cross-validation gets more
data.
After collecting all rij values, Wu et al. (2004) propose several approaches to
obtain pi , i. In LIBSVM, we consider their second approach and solve the following
optimization problem.
k
1XX
(rji pi rij pj )2
2 i=1 j:j6=i
min
p
subject to
k
X
pi 0, i,
pi = 1.
(44)
i=1
1 T
p Qp,
2
min
p
where
(P
Qij =
2
s:s6=i rsi
rji rij
if i = j,
if i =
6 j.
Q e
eT 0
p
0
=
,
b
1
(45)
where e is the k 1 vector of all ones and 0 is the k 1 vector of all zeros.
Instead of solving the linear system (45) by a direct method such as Gaussian
elimination, Wu et al. (2004) derive a simple iterative method. Because
pT Qp = pT Q(be) = bpT e = b,
the optimal solution p satisfies
(Qp)t pT Qp = Qtt pt +
X
j:j6=t
Qtj pj pT Qp = 0,
t.
(46)
Algorithm 3
1. Start with an initial p satisfying pi 0, i and
Pk
i=1
pi = 1.
2. Repeat (t = 1, . . . , k, 1, . . .)
pt
X
1
[
Qtj pj + pT Qp]
Qtt j:j6=t
(47)
normalize p
until Eq. (45) is satisfied.
Eq. (47) can be simplified to
pt p t +
1
[(Qp)t + pT Qp].
Qtt
When k (the number of classes) is large, some elements of p may be very close to
zero. Thus, we use a more strict stopping condition by decreasing the tolerance by a
factor of k.
Next, we discuss SVR probability inference. For a given set of training data
D = {(xi , yi ) | xi Rn , yi R, i = 1, . . . , l}, we assume that the data are collected
from the model
yi = f (xi ) + i ,
where f (x) is the underlying function and i s are independent and identically distributed random noises. Given a test data x, the distribution of y given x and D,
P (y | x, D), allows us to draw probabilistic inferences about y; for example, we can
estimate the probability that y is in an interval such as [f (x), f (x)+]. Denoting
f as the estimated function based on D using SVR, then = (x) y f(x) is the
out-of-sample residual (or prediction error). We propose modeling the distribution of
based on cross-validation residuals {i }li=1 . The i s are generated by first conducting a five-fold cross-validation to get fj , j = 1, . . . , 5, and then setting i yi fj (xi )
for (xi , yi ) in the jth fold. It is conceptually clear that the distribution of i s may
resemble that of the prediction error .
32
1 |z|
e .
2
Assuming that i s are independent, we can estimate the scale parameter by maximizing the likelihood. For Laplace, the maximum likelihood estimate is
Pl
|i |
= i=1
.
l
Lin and Weng (2004) point out that some very extreme i s may cause inaccurate
estimation of . Thus, they propose estimating the scale parameter by discarding
i s which exceed 5 (standard deviation of the Laplace distribution). For any new
data x, we consider that
y = f(x) + z,
where z is a random variable following the Laplace distribution with parameter .
In theory, the distribution of may depend on the input x, but here we assume
that it is free of x. Such an assumption works well in practice and leads to a simple
model.
Parameter Selection
To train SVM problems, users must specify some parameters. LIBSVM provides a
simple tool to check a grid of parameters. For each parameter setting, LIBSVM obtains
cross-validation (CV) accuracy. Finally, the parameters with the highest CV accuracy
33
Figure 2: Histogram of i s and the models via Laplace and Gaussian distributions.
The x-axis is i using five-fold cross-validation and the y-axis is the normalized number
of data in each bin of width 1.
are returned. The parameter selection tool assumes that the RBF (Gaussian) kernel
is used although extensions to other kernels and SVR can be easily made. The RBF
kernel takes the form
2
K(xi , xj ) = ekxi xj k ,
(48)
34
Figure 3: Contour plot of running the parameter selection tool in LIBSVM. The data
set heart scale (included in the package) is used. The x-axis is log2 C and the y-axis
is log2 .
Figure 3.
10
Conclusions
When we released the first version of LIBSVM in 2000, only two-class C-SVC was
supported. Gradually, we added other SVM variants, and supported functions such
as multi-class classification and probability estimates. Then, LIBSVM becomes a
complete SVM package. We add a function only if it is needed by enough users. By
keeping the system simple, we strive to ensure good system reliability.
In summary, this article gives implementation details of LIBSVM. We are still
actively updating and maintaining this package. We hope the community will benefit
more from our continuing development of LIBSVM.
35
Acknowledgments
This work was supported in part by the National Science Council of Taiwan via the
grants NSC 89-2213-E-002-013 and NSC 89-2213-E-002-106. The authors thank their
group members and users for many helpful comments. A list of acknowledgments is
at http://www.csie.ntu.edu.tw/~cjlin/libsvm/acknowledgements.
References
B. E. Boser, I. Guyon, and V. Vapnik. A training algorithm for optimal margin
classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning
Theory, pages 144152. ACM Press, 1992.
C.-C. Chang and C.-J. Lin. Training -support vector classifiers: Theory and algorithms. Neural Computation, 13(9):21192147, 2001.
C.-C. Chang and C.-J. Lin. Training -support vector regression: Theory and algorithms. Neural Computation, 14(8):19591977, 2002.
C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vector machines. ACM
Transactions on Intelligent Systems and Technology, 2:27:127:27, 2011. Software
available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
P.-H. Chen, C.-J. Lin, and B. Scholkopf. A tutorial on -support vector machines.
Applied Stochastic Models in Business and Industry, 21:111136, 2005. URL http:
//www.csie.ntu.edu.tw/~cjlin/papers/nusvmtoturial.pdf.
P.-H. Chen, R.-E. Fan, and C.-J. Lin. A study on SMO-type decomposition methods
for support vector machines. IEEE Transactions on Neural Networks, 17:893908,
July 2006.
pdf.
URL http://www.csie.ntu.edu.tw/~cjlin/papers/generalSMO.
36
BDVal: repro-
39