Fifth International Conference on Fuzzy Systems and Knowledge Discovery
Fuzzy Theory Based Support Vector Machine Classifier∗
Xuehua Li
School of Applied Mathematics,
University of Electronic Science and
Technology of China, 610054, P.R. China
leesoftcom@gmail.com
Abstract
Support vector machine (SVM) has become a popular
tool in the area of pattern recognition, combining support
vector machines with other theories has been proposed as
a new direction to improve classification performance. This
paper applies fuzzy theory to support vector machines for
classification. In the first phase, a fuzzy support vector ma-
chine is proposed for the classification of real-world data
with noise, fuzzy membership to each data point of SVM
and reformulates the SVM such that different input points
can make different contributions to the each class. In the
second phase, the SVM’s kernel’s parameters are calculated
by the kernel’s parameters evaluation function. To investi-
gate the effectiveness of the proposed fuzzy support vector
machine classifier, it is applied to the given dataset, the ex-
perimental results confirm the superiority of the presented
method to the traditional SVM classifier.
Keywords: Support vector machine, Kernel method, Fuzzy
theory, Kernel parameters evaluation function.
1. Introduction
Support vector machine is learning system that uses a
hypothesis space of linear function in high dimensional
feature space[1], which has been recently introduced for
solving pattern recognition and function estimation prob-
lems. The theory of SVM is based on the idea of structural
minimization, which shows that the generalization error is
bounded by the sum of the training set and a term depend-
ing on the Vapnik-Chervonenkis dimension[2]. By mini-
mizing this bound, high generalization performance can be
achieved. Moreover, unlike other machine learning meth-
ods, SVM generalization error is not related to the prob-
lem’s input dimensionality. This explains why SVM can
have good performance even in high dimensional problem.
∗ Supported by Foundation of National Natural science (No.10671030)
978-0-7695-3305-6/08 $25.00 © 2008 IEEE
DOI 10.1109/FSKD.2008.440
Lan Shu
School of Applied Mathematics,
University of Electronic Science and
Technology of China, 610054, P.R. China
shul@uestc.edu.cn
SVM maps input data into high dimensional characteristic
space in which it constructs an optimal separating hyper-
plane. In many applications it provides high generalization
ability and overcomes the overfitting problem experienced
by the other leaning techniques such as ANN.
However, the practical input data are usually contami-
nated by noise which caused the disturbance and measure-
ment, and data close to the class boundary is most affected
by noise, the standard SVM formulation is not robust to
noise. Some important qualitative information about the in-
put data can’t apply well to the SVM. In order to realize
practical application, we proposed to include fuzzy theory
to improve SVM classify ability. As we know, fuzzy logic
is powerful tool to deal with uncertain, nonlinear, ill-posed
problem. Fuzzy logic is appropriate to present the quali-
tative and ambiguous knowledge. The inference model of
fuzzy logic is similar with that of the human thought. So
combinating the SVM with fuzzy logic is a kind of promis-
ing method[3]. Recently, fuzzy SVM have been attracting a
lot of interest and there have many papers to explore fuzzy
SVM for classification problem[3,4,5], and fuzzy SVM has
been shown to be extremely successful, but few papers dis-
cuss fuzzy SVM with kernel, hence, it will be not effica-
cious to nonlinear classification problem.
Kernel method is the most important technique for SVM
to deal with nonlinear problem, which applies SVM to learn
non-linear relations with a linear machine[1], and the attrac-
tion of kernel method is that the learning algorithms and
theory can largely be decoupled from the specifications of
the application area, which must simply be encoded into
the design of an appropriate kernel function. Hence, the
problem of choosing a suitable kernel for SVM is more im-
portant. There have several well-known kernels, such as
Radial Basis Function (RBF) kernel, which using in many
papers, but another problem is present, how many the val-
ues of the kernel’s parameters, and how to evaluate effect
of SVM with the kernel which has been given the selected
parameters.
In this paper, to overcome these problems, we propose
600

fuzzy support vector machine (FSVM) and kernel’s pa-
rameters evaluation function. Firstly, we propose FSVM
which improves the performance of traditional SVM by in-
cluding fuzzy membership to each input value to show the
membership degree of the input values to different classes,
and FSVM uses kernel to deal with nonlinear classification
problem in section 3. Secondly, a kernel’s parameters eval-
uation function build by fuzzy membership and the learn-
ing algorithm are given in section 4. Thirdly, we give a set
of experiments, discussions and comparisons for the SVM
which combining with fuzzy theory in section 5. Finally,
conclusions are drawn in section 6.
2. Summary of support vector machines
The original support vector machine can be character-
ized as a powerful learning algorithm based on recent ad-
vances in statistical learning theory[6,8]. SVM is a learning
system that uses a hypothesis space of linear functions in a
high-dimensional space, trained with a learning algorithm
from optimization theory that implements a learning bias
derived from statistical learning theory. SVM uses a lin-
ear model to implement nonlinear class boundaries by map-
ping input vectors nonlinearly into a high-dimensional fea-
ture space using kernels. SVM has recently become one of
the most popular tools for machine learning and data mining
and can perform both classification and regression.
Let S = {(x1 , y1 ), (x2 , y2 ), ......., (xn , yn )} be a training set
with input data, where xi ∈ ℜn is the training data and cor-
responding binary class labels yn ∈ {−1, 1}. Let the weight
and the bias of the separating hyperplane be w and b, re-
spectively, and the SVM classifier is
f (xi ) = sgn(wϕ(xi ) + b)
where ϕ is a nonlinear function, which maps x the input
space into a feature space, To separate the data linearly in
the feature space, the decision function satisfies the follow-
ing conditions and the optimization problem is
Minimize kwk2 =< w, w >
subject to yi [wϕ(xi ) + b] ≥ 1,
i = 1, 2, ......., n
2
xi − x̄ j
The objective of SVM is to maximize the margin of sep-
aration and minimize the training errors. The problem can
then be transformed into the following Lagrange formula-
tion
n
1
n th of d-dimensional measured data, x̄ j is the d-dimension
maximize L(α) = ∑ αi − 2 ∑ yi y j αi α j K(xi , x j )
i=1 i, j=1
n (3)
subject to ∑ yi αi = 0
i=1
αi ≥ 0, i = 1, 2, .....n
where K(xi , x j ) = ϕ(xi ), ϕ(x j ) is a kernel function
and satisfies the Mercer theorem. The Karush-Kuhn-
Tucker(KKT) complementarity conditions provide useful
information about the structure of the solution. The condi-
tions state that the optimal solutions α ∗ , w∗ , b∗ must satisfy
αi∗ [y(w∗ ϕ(xi ) + b∗ ) − 1] = 0, i = 1, 2, ....n. (4)
where the αi∗ are the solutions of the dual problem and
are non-zero only for a subset of vectors xi∗ called support
vectors. Then the resulting SVM for function estimation
becomes
m
f (x) = sgn( ∑ αi∗ yi K(x, xi∗ ) + b) (5)
i=1
where m is the number of support vectors. SVM is a new
paradigm of the learning system. SVM technique is a pow-
erful widely used technique for solving supervised classifi-
cation problems due to its generalization ability. In essence,
SVM classifiers maximize the margin between training data
and the decision boundary (optimal separating hyperplane),
which can be formulated as a quadratic optimization prob-
lem in a feature space.
3 Fuzzy support vector machines
In this paper, by extending and developing soft margin
algorithm, we introduce the fuzzy memberships of train-
ing points in the hyperplane definition, by exploring the
fuzzy information inherent in the training set[9]. Let us
consider the aforementioned binary training set S. We use
fuzzy memberships [µi− , µi+ ] to weight the importance of
(1) the training point xi in the hyperplane computation, where
µi− is the membership value denoting the point xi toward
the class where yi = −1, and µi+ means the membership
value to the positive class. At first, we get all cluster centers
of training set S by Fuzzy C-means (FCM) algorithm[10],
the objective of FCM algorithm is to minimize the Fuzzy
C-Means cost function formulated as
(2)
t n
∑ ∑ (µi j )m
J= (6)
j=1 i=1
where m is any real number greater than 1, µi j is the de-
gree of membership of xi in the cluster t and µi j ∈ [0, 1], ∀i =
t
1, ......n, j = 1, ......t, ∑ µi j = 1, ∀i = 1, ......, n, xi is the i-
j=1
center of the cluster, and || ∗ || is any norm expressing the
similarity between any measured data and the center.
Let X̄ + = (x̄1+ , ......x̄l+ ) is the cluster centers of the class
{yi = +1} and X̄ − = (x̄1− , ......x̄m − ) is the cluster centers of
601
 n
the negative class. We denote u+ ki is the degree of member-
ship of xi in the cluster xk+ , the µki+ is defined as follows ∑ αi sgn(µi+ − µi− ) = 0 (14)
i=1

1 xi −x̄
+ ∂ L(w, b, α, β , ξi+ , ξi− )
µki+ =e −2( σ k )2 k = 1, 2....., l. (7) =0→
∂α
n
Then define the membership of xi in the class{yi = +1},
µi+ as
∑ sgn(µi+ − µi− )(wxi + b + µi+ ξi+ − µi− ξi− ) − 1 = 0
i=1
(15)
µi+ = max(µ1i+ , µ2i+ , ........, µli+ ) (8)
∂ L(w, b, α, β , ξi+ , ξi− )
The membership of xi in the class{yi = −1}, µi− is de- =0→
∂β
fined as µi+
n

1 xi −x̄
−
− 2 ( σ k )2
∑ sgn(µi+ − µi− )(µi+ ξi+ − µi− ξi− ) = 0 (16)
− i=1
µki = e k = 1, 2....., m. (9)
According to KKT conditions, the solutions αi∗ to
µi = max(µ1i , µ2i , ........, µli− )
− − −
(10)
Eq.(15) satisfy the following formulae
We give a set S of training points: S =
{(x1 , µ1− , µ1+ ), (x2 , µ2− , µ2+ ), ......., (xn , µn− , µn+ }. The
optimal hyperplane problem is regarded as the solution to αi∗ [sgn(µi+ − µi− )(w∗ xi + b∗ + µi+ ξi+ − µi− ξi− ) − 1] = 0
ξi+ (Cµi+ − αi∗ ) = 0, ξi− (Cµi− − αi∗ ) = 0
Minimize 1
< w, w > + (17)
2
n
C ∑ sgn(µi+ − µi− )(µi+ ξi+ − µi− ξi− )
i=1 4 The evaluation function of SVM’s kernel
subject to sgn(µi+ − µi− )(wxi + b) ≥ parameters
1 − sgn(µi+ − µi− )(µi+ ξi+ − µi− ξi− )
ξi+ ≥ 0, ξi− ≥ 0,C ≥ 0 , i = 1, 2, .....n.
(11)
where C defines how much weight is given to the minimiza-
tion of the slack vector as compared to the weight vector[7].
Hence, the following Lagrange function could be obtained

L(w, b, α, β , ξi+ , ξi− ) =

1
2 < w, w >
n
+C ∑ si (µi+ ξi+ − µi− ξi− )
i=1
n
− ∑ αi [si (wxi + b + µi+ ξi+ − µi− ξi− ) − 1]
i=1
n
− ∑ βi si (µi+ ξi+ − µi− ξi− )
i=1
Figure 1. SVM with RBF Kernel ( σ = 0.1,C = 5)
αi ≥ 0, βi ≥ 0, i = 1, 2, .....n (12)
where si = sgn(µi+ − µi− ), and < w, w >=
n Kernel method is demonstrated to be able to extract
∑ sgn(µi+ − µi− )sgn(µ + −
j − µ j )αi α j K(xi , x j ). the complicated nonlinear information embedded on the
i, j=1
data-set. Kernel function is a nonlinear mapping from in-
The conditions for optimality are as follows put space X ⊆ ℜn onto feature space Z ⊆ ℜN , ϕ : X ⊆
ℜn → Z ⊆ ℜN [11]. Kernel method provides a powerful
∂ L(w, b, α, β , ξi+ , ξi− ) and principled way of detecting nonlinear relations using
=0→
∂w well-understood linear algorithms in an appropriate feature
n space. The approach decouples the design of the algorithm
w − ∑ αi wxi sgn(µi+ − µi− ) = 0 (13) from the specification of the feature space. In real Appli-
i=1 cations, the first problem is selection of the SVM’s ker-
∂ L(w, b, α, β , ξi+ , ξi− ) nel, which amounts to learning model selection. Any prior
=0→ knowledge we have of the problem can help in selecting
∂b

602
the parameters. For most classes of kernels, for example
polynomial kernel or RBF kernel, it is always possible to
find kernel parameters for which the data become separa-
ble. However, forcing separation of the data can easily lead
to overfitting, particularly when noise is present in the data.
In the other hand, the support points contain the informa-
tion necessary to reconstruct the hyperplane, in general, the
fewer the number of support vectors the better generaliza-
tion can be expected, but too few support vectors will reduce
the accuracy of SVM’s classifier.
Fig. 1 and Fig. 2 show SVM for learning a data set
using RBF kernels with different values of σ and C, we will
find that the two cases the classification of the training set
is consistent, and the larger value of σ ,the smaller number
of support vectors, the more errors of classification will be
present.
Figure 2. SVM with RBF Kernel ( σ = 2,C = 10)
In this section, we dispose a function to evaluate the pa-
rameters of the kernel. In SVM, the solution to the prob-
lem is only dependent on a subset of training data points
which are referred to as support vectors. Using only sup-
port vectors, the same solution can be obtained as using all
the training data points. Then, the support points contain
all the information necessary to reconstruct the hyperplane,
even if all of the other points were removed the same max-
imal separating hyperplane would be found for the remain-
ing subset of the support vectors. The support vectors the
solution of the Lagrange function which contains kernel’s
parameters, hence, the relationship between SVM classifier
and parameters is present.
The objective of SVM is to maximize the margin of sep-
aration, and the solution of the optimization problem(3),
which is equivalent to the hyperplane in the feature space
implicitly defined by the kernel. The SVM’s geometric mar-
gin is
n
γ =< w∗ , w∗ >= yi y j αi∗ α ∗j K(xi , x j )
∑ the unclassifiable region is much higher than those of SVM,
i, j=1
Where xi , x j are the support vectors. In general, one prob-
lem with the SVM maximize margin suggested is the choice
of parameters, typically a range of values must be tried be-
fore the best choice for a particular training set can be se-
lected. There are many support vectors produced by differ-
ent parameters, let S={x1 , x2 , ...........xl } is the intersection
of all support vectors, the margin function
l
w(θ ) = ∑ yi y j ᾱi ᾱ j K(xi , x j ), xi , x j ∈ S (19)
i=1
Where θ is the kernel’s parameter vector, and ᾱi is aver-
age value of all the optimization problem solution αi of the
support vectors xi . Then we defined M is the maximize of
w(θi ), M = max(w(θi )), where θi is the i-th parameter of
all the training parameters. Then define µ(θi ) is the margin
membership of parameter θi as
w(θi )
µ(θi ) = (20)
M
The kernel’s parameters evaluation function is defined
1
g(θi ) = exp[− (µ(θi ) − 1)2 ] (21)
2
Where g(θi ) → [0, 1], which estimates parameter θi ef-
fect to the SVM classifier. Thus, the parameter θi is com-
puted by the evaluation function which is most seemly value
of the kernel’s parameters.
5 Performance Evaluation
This section presents a set of experiments that were car-
ried out by using the Allbp database which be obtained from
UCI repository of machine learning database[12], The rea-
son for using this dataset is that because it is very commonly
used among the other classification systems we can com-
pare our system with them for thyroid diagnosis problem,
we select Allbp dataset contains 972 samples, each consist-
ing of 29 attributes (features), this is a three-class classifi-
cation problem. We choose 364 samples as the training set,
and 608 samples as the test set. As stated earlier the RBF
kernel was used due to its better performance.
In order to illustrate the performance of the kernel’s pa-
rameters evaluation function, Fig 3 shows the kernel’s pa-
rameters evaluation value by the parameter σ from 0.001
to 2 and C from 0.1 to 20. We get the maximal evaluation
value when σ =0.16 and C=5.2. Table I shows the compare
training accuracy and testing accuracy of SVM with FSVM,
and table I shows the effect of the different selection of ker-
nel’s parameter, Table II shows the Comparisons CPU times
and support vectors number of SVM and FSVM. From table
(18) I, we can easily conclude that the test accuracy of FSVM in
603
 function improves the SVM classifier accuracy.
Table 1. Comparisons of two SVM’s classifi-
cation accuracy References
Method Parameter Training AC Testing AC
LS-SVM (0.01, 1) 91.2% 83.2%
[1] N. Cristianni and J. Shawe-Talyor, An introduction
LS-SVM (0.16, 5.2) 95.5% 89.2%
to support vector Machines, Cambridge Uninversity
LS-SVM (1, 10) 87.9% 79.3%
Press, 2000.
FSVM (0.01, 1) 92.1% 84.7%
FSVM (0.16, 5.2) 98.6% 92.5% [2] V.N. Vapnik, An Overview of Statistical Learning The-
FSVM (1, 10) 88.4% 82.6% ory, IEEE Transactions on Neural Networks, 10(5):
988-999, 1999.
[3] C.F. Lin, and S.D. Wang, Fuzzy Support Vector Ma-
from table II, the SVM classifier is more effective by us- chines, IEEE Transactions on Neural Networks, 13(3):
ing kernel’s parameters evaluation function. As compared 466-471, 2002.
to SVM and the classification results, the SVM using fuzzy
theory can not only improve the SVM classifier accuracy, [4] Danisuke Tsujinishi and Shigeo Abe, Fuzzy least
but also get optimized kernel parameters learning algorithm. squares support vector machines for multiclass prob-
lems, Neural Nehuorks, 16: 785-792, 2003.
[5] T. Inoue and S. Abe, Fuzzy support vector machines
for pattem classification, International Joint Confer-
ence on Neural Networks, 2: 1449-1454, July, 2001.
[6] V.N. Vapnik, Statistical Learning Theory, John Wiley
and Sons, New York, 1998.
[7] Shawe Talyor J., Cristianini N. Kernel Methods for
Pattern Analysis. Cambridge, Cambridge Uninversity
Press, 2004.
[8] V.N. Vapnik. The nature of statistical learning theory.
NY: Springer-Verlag, 1995.
[9] C.F. Lin and S.D. Wang, Training algorithms for
fuzzy support vector machines with noisy data, Pattern
Figure 3. Parameters evaluation value Recognition Letters, 25(10): 1647-1656, 2004.
[10] C.Y. Yang, Support vector classifier with a fuzzy-
value class label, Lecture Notes in Computer Science,
6 Conclusion Springer-Verlag, Berlin, 3173: 506-511, 2004.
[11] B. Schölkopf and A. Smola, K.R. Müller, Learn-
In this paper, we proposed fuzzy support vector machine
ing with Kernels: Support Vector Machines, Regular-
for classification. The proposed FSVM resolves unclassi-
ization, Optimization and Beyond, Cambridge, MIT
fiable regions caused by conventional support vector ma-
Press, 2002.
chines. We apply a fuzzy membership function to each data
point and reformulate the SVM such that different input [12] C.L. Blake and C.J. Merz UCI repository of ma-
points can make different contributions to the learning of chine learning databases. Dept. Inform. Comput. Sci.,
Lagrange function. On the other hand, the use of kernel’s Univ. California, Irvine, CA. [Online]. Available:
parameters evaluation function provides the seemly param- http://www.ics.uci.edu/ mlearn/MLRepository.html.
eters of SVM’s kernel. The result of experiments indicates
that the proposed method enhances the SVM in reducing
the effect of noises in the application. FSVM is suitable for
applications in which data points have modeled characteris-
tics. It demonstrates the superiority of the method of FSVM
over standard SVM, and the kernel’s parameters evaluation

604