3rd IEEE International Conference on "Computational Intelligence and Communication Technology" (IEEE

(IEEE-CICT 2017)

Classification
sification Optimization using PSO
PSO--SSO based
Support Vector Machine
Lokesh P. Gagnani K H Wandra Hitesh Chhinkaniwala
Department of Computer Dean, Department of Computer
Engineering. FTE, Engineering,
C U Shah University, C U Shah University, AIIE,
Wadhwan, Gujarat, INDIA Wadhwan, Gujarat, INDIA Ahmedabad Gujarat, INDIA
Ahmedabad,
gagnani.lokesh@gmail.com

Abstract—Classification is one of thehe widely used technique UCI datasets.. Finally Section V summarizes and concludes the
for data mining of dataset and is done using soft com computing research in this paper.
approach. Here a novel method called SSO-ELS ELS is pproposed for
classification of datasets. In this method there
here is hybridization of
Simplified Swarm Optimization (SSO) with ELS (Exchange II. BACKGROUND
Local Search),, Particle Swarm Optimization (PSO) and Support
Vector Machines(SVM) approach. This is done to resolve the A. Support Vector Machine (SVM)
issue of selection of hyper parameters in SVM. The selection of Support Vector Machine (SVM) is a supervised
hyper parameters in SVM plays a crucial rule which is done by classification technique in Machine Learning. In layman's
the PSO-SSO approach. This approach has two phases: IIn first term, it is for finding the hyper plane which may be line, plane
phase best initial parameters of SVM are calculated using SSO or hyper plane in accordance with dimensionality which
with ELS approach and then the best parameters are fed into separates the two classes with maximum margin. This is
SVM using PSO in second phase. Brief review of classification
illustrated in Fig 1. The data points that kind of support this
methods is discussed. Experiments on UCI datasets indicate that
hyper plane on either sideses are called as support vectors.
the proposed SSO-PSO-SVM achieves hieves better res
results than CS-
PSO-SVM with respect to classification accuracy and F F-measure.

Keywords—Soft
Soft Computing, Support Vector Machine, Par
Particle
Swarm Optimization, Simplified Swarm Optimization,parameter
optimization

I. INTRODUCTION
Data Mining is the approach of extracti
extraction of useful
patterns from datasets using approaches like classification,
clustering, association rule mining, regression, etc
etc. In other
words data mining is to analyze data from different
perspectives and summarize into useful infor
information[12]. Of Figure 1: Hyperplane in SVM
these various approaches classification has more importance in
This linearly separable units comprise the Hard margin.
Support Vector Machines (SVM). Classification in data
However for non-linear
inear separable hard margin fails and soft
mining assigns data in groups to target classes. The main
margin is used. For non-linear
linear kernels are used that converts
objective of classification is to accurately predict the target
them into linear separable in some high dimensional plane.
plane
class for each case in the data.
A kernel is a similarity function that corresponds to an
Swarm Intelligence has been recently emerged branch of inner product in some featured space.
spa Various kernels for non-
Soft Computing that includes populationn-based stochastic linear transformation are given below:
optimization techniques. Swarm Intelligence was introduced
by Beni and Wang in 1989[4]. RBF Kernel (Gaussian Kernel)
The RBF kernel of 2 samples X and X’ as feature vectors in
The paper is organized as: Section II gives brief overview some input space is represented as:
of SVM, PSO and SSO methods,, and provides a rreview of
various classification methods.. Section III explains in detail || ||
the proposed SSO-ELS with PSO novel classification 𝐾(𝑥, 𝑥 ) = 𝑒𝑥𝑝 − (1)
approach. This includes the SSO algorithm with proposed ELS Sometimes it is represented as::
strategy. Section IV describe the experiment
experiments and results on

978-1-5090-6218-8/17/$31.00
8/17/$31.00 ©2017 IEEE

1
 3rd IEEE International Conference on "Computational Intelligence and Communication Technology" (IEEE-CICT 2017)

𝐾(𝑥, 𝑥 ) = 𝑒𝑥𝑝(−𝛶 ||𝑥 − 𝑥′|| ) (2) 11. Jiri Krupka RFC Conditional rules as IF-THEN
(2014)[6] done by RST and Mamdani
interference with fuzzy
Polynomial kernel controller.
For degree-d polynomial ad x and y as feature vectors in input 12. B. AGFS Rule generation with GA and
Dennis(2014)[1] then classification through
space, the polynomial kernel is represented as: fuzzy system
13 Xiaoyong Liu PSO-SVM – A CS based approach for
(2014)[13] CS parameter optimization of
𝐾(𝑥, 𝑦) = (𝛼𝑥 𝑦 + 𝑐) (3) SVM & PSO is applied to
continue SVM training.
Compared with GA-SVM and
Hyperbolic Tangent (Sigmoid) Kernel PSO-SVM
It is represented as:
𝐾(𝑥, 𝑦) = 𝑡𝑎𝑛ℎ(𝛼𝑥 𝑦 + 𝑐) (4)
C. Cuckoo Search
A novel approach developed by Yang[14] which has
Here the 2 adjustable parameters are the alpha and intercept
behavior similar to obligate blood parasitism. It lays the eggs
denoted as 𝛼 & 𝑐 respectively. Default value of 𝛼 with N data in the nests of other species-host birds. Direct conflict with
dimensions can be 1/N. cuckoos of intruder type is possible for some host birds. On
discovering alien eggs a host bird with simply leave its nest or
B. Brief Review of Classification Techniques throw them. Also some chosen host species named New
World Tapera are specialized in doing imitation in eggs
Brief review of various classification techniques along patterns and colors specially the that the female parasitic
with the merits is illustrated in Table 1. cuckoos.

Here Cuckoo Search via Levy Flights is taken for

TABLE I :REVIEW OF CLASSIFICATION TECHNIQUES comparison with the proposed approach. The Levy flight is
essentially a random walk with random step using a Levy
distribution:
No Author Approaches Brief Overview of
Approach/Merit
1. Ziqiang Wang PSO Less operations & parameters
Levy~u=t-λ (1< λ ≤ 3) (5)
(2007) [16] than GA. Compared with ant-
Miner and OCEC
2. Lale O zbakır DIFACONN DE is used in training and III. PROPOSED SSO-ELS WITH PSO
(2009)[7] ACO in rule extraction.
3. Mete ÇELİK ABC-Miner ABC to mine classification
(2011)[9] rules. Compared with c4.5 The optimal parameters are found from SSO-ELS
and PSO (Simplified Swarm Optimization with Exchange Local Search
4. H. Kasiri FRENGA Genetic Fuzzy System (GFS) Strategy) in which range of parameters is taken as Lower
(2011)[5] for rule extraction from Bound and Upper Bound. These optimal values taken as input
MLPNN. continuous and in PSO (Particle Swarm Optimization) for finding best
enumerates
attributes
parameters and that are entered into SVM (Support Vector
5 Pavel Kromer GA evolve Fuzzy classifier are evolved Machines) for classification accuracy. Two hyper parameters
(2011)[10] fuzzy by GA of SVM are the C and ɛ.
classifier
6 Yao Liu (2012) SSO A novel global optimization Considering various parameters of SVM C and ɛ are taken
[15]
method inspired by simplified into account. They are named as penalty factor and insensitive
Swarm Optimization. parameter respectively. The smaller the penalty (less C), the
7 Changseok Bae SSO-ELS A novel approach based on lesser affect on empirical risk and the approximation model is
(2012)[3] Simplified Swarm
Optimization with ELS simpler. Further the larger the penalty (large C),the more
strategy affect on empirical risk and the model is complex. The ratio of
8. C. F. F. Carraro Fuzzy GA automatic confidence interval and empirical risk is affected by C. It takes
(2013)[2] rule adjustment of the membership values as [1,10000]. The larger the insensitive parameter, the
extraction functions is done by GA
form SVM precision decreases and support vectors are less. It takes
9. Soumadip Neuro Fuzzy MLPBPN and fuzzy set values as [0.0001, 0.1].
Ghosh theory approach are .
(2014)[11] combined. Symbolic and
numeric
10. Marconi de MOGP Rule extraction by mo and
In this paper the values are restricted in intervals:
Arruda Pereira optimization by ɛ ∈ [0.001, 1] , C ∈[1, 15].
(2014)[8] GA.Compared with DT, RBF,
SVM

2
 3rd IEEE International Conference on "Computational Intelligence and Communication Technology" (IEEE-CICT 2017)

STEPS FOR SSO:

𝑉 (𝑖 + 1) = 𝜔. 𝑉 (𝑖) + 𝐶1. ∅ . 𝑃 − 𝑋 (𝑖) + 𝐶2. ∅ . 𝐺 −
(1) Initialization of parameters.
m is swarm size. Cw,Cp,Cg are predetermined constants, 𝑋 (𝑖) (6)
maxGen is Maximum generation and maxFit is maximum
fitness value. 𝑋 (𝑖 + 1) = 𝑋 (𝑖) + 𝑉 (𝑖 + 1) (7)
(2) Generate and initialize pbest and gbest with random
poistion(x) The proposed SSO-ELS with PSO in SVM can be described
(3) Fitness value for each particle is calculated as shown in Fig 2.
(4) Local best and global best are updated

SSO
Find Initial value of C & ε
(5) Random Number Generation by SSO method
Check 3 conditions:
0 ≤ R ≤ Cw
0 ≤ R ≤ Cp Generate initial particles
0 ≤ R ≤ Cg
Replace the original value by generating random new value Evaluate the fitness value
(4) Repeat until termination criteria is met.
Evaluation of fitness values, determine the
STEPS FOR ELS: local best and global best of each particle

PSO
(1) Take initial local search time that will be used for Update the velocities & positions until
gbest. fitness value converges

(2) Calculate the pbest and gbest.

(3) Randomly select one attribute and is called
Termination
exchangeAttribute Criteria?
(4) Randomly select LowerBound(x) and
UpperBound(x) from exchangeAttribute Px and Py
Best value of C and ε fed into SVM
(5) Update exchangeAttribute value of target particle(Pi)
with x and y
Training & Testing of dataset using

SVM
(6) Evaluate fitness value (Pi) SVM Classifier
(7) Update pbest position as per comparison of Fitness
value of Pi and pbest Accuracy, F-Measure, Precision & Recall

(8) Update gbest position as per comparison of Fitness

value of Pi and gbest Figure 2: Proposed SSO-PSO with SVM
(9) End local search
IV. RESULT ANALYSIS
The process will be repeated until all particles have completed A. Experimental Setup
the local search strategy.
The hybrid SSO-PSO with SVM was programmed in Java in
PSO. PSO is a swarm intelligence technique mainly for Netbeans 8.0 platform and run on machine with Core i3
optimization purpose. A particle is represented as a single processor, 2.20 Ghz, 6GB RAM.
solution in PSO search space. Fitness function is indicated by
fitness values of all particles and velocities by the flying of B. Dataset Description
particles. In every step, a particle encounters with two values. Two datasets namely Cancer and Heart Disease from UCI[17]
Initial best solution is termed as local and other as global are used. The details of it are shown in Table II:
based on minimization optimization.
TABLE II: DATASET DESCRIPTION
PSO is widely used optimization technique in various
applications due to less parameters and simplicity than other Name Rows Attributes Class #Tr #Test
optimization techniques in its category. Particles fitness value Attribute Rows rows
can be evaluated by the input values which are particle Cancer 699 10 {2,4} 550 149
Heart 270 13 {1,2} 190 80
positional coordinate. Further Vi and Xi are the velocities and
position values of particles respectively. Then the particles
updates their velocity and position using the equations below: Tr is training rows and Test is testing rows.

3
 3rd IEEE International Conference on "Computational Intelligence and Communication Technology" (IEEE-CICT 2017)

C. Simulation Results and Analysis Heart 75 100 75 86 93 96 94 95

The performance is compared between existing CS-PSO-SVM V. CONCLUSION
and proposed SSO-PSO-SVM by running models for several
iterations. The datasets used are normalized to numerical The results conclude that the novel classification SSO-PSO-
values by manual substitution. Accuracy is calculated by the SVM approach is better than the existing approach CS-PSO-
confusion matrix. SVM with respect to F-measure and classification accuracy.
This can inferred from results shown in Table V.
The classification metrics for evaluation are the described as:
ACKNOWLEDGMENT
(a) Confusion Matrix: It is s table that present the predicted
results of a classifier. Also called as a contigency table. The authors are highly obliged to Dean, Vice
Chancellor and faculties of Research Development &
TABLE III :CONTIGENCY TABLE Innovation Council at C U Shah University, Wadhwan, India.
Positive Negative
Positive TP FN REFERENCES
Negative FP TN
[1] B.Dennis ,S.Muthukrishnan ,“AGFS: Adaptive Genetic Fuzzy System
where TP=True Positive,FP=False Positive,FN=False for medical data classification”, Elsevier, 2014, pp. 242-252.
Negative,TN=True Negative. [2] C. F. F. Carraro and M. Vellasco, R. Tanscheit, “A Fuzzy-Genetic
(b) Precision: Precision can be thought of as a measure of a System for Rule Extraction from Support Vector Machines”, 2013,
IEEE, pp. 362-367.
classifiers exactness. A low precision can also indicate a large [3] Changseok Bae, Wei-Chang Yeh, Noorhaniza Wahid, Yuk Ying
number of False Positives. Chung, Yao Liu, "A New Simplified Swarm Optimization (SSO)
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑇𝑃 ⁄(𝑇𝑃 + 𝐹𝑃) (8) Using Exchange Local Search Scheme ", International Journal of
(c) Recall: Recall can be thought of as a measure of a Innovative computing, Information and Control (IJICIC), Vol. 8, No.
6, pp. 4391-4406, 2012.
classifiers completeness. A low recall indicates many False [4] G. Beni, J. Wang, Cellular Robotic Systems, 1989.
Negatives. Also referred as Sensitivity. <http://en.wikipedia.org/wiki/Swarm_intelligence>
𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑇𝑃 ⁄(𝑇𝑃 + 𝐹𝑁) (9) [5] H. Kasiri, M. Saniee Abadeh, H. Momeni, A. R. Motavalian, “Fuzzy
(d) F-Measure: the F1 score conveys the balance between the Rule Extraction from a Trained Artificial Neural Network using
Genetic Algorithm for WECS Control and Parameter Estimation”,
precision and the recall. 2011, IEEE, pp. 635-639.
( ∗ ) [6] Jiri Krupka, Pavel Jirava,“Rough-Fuzzy classifier modeling using data
𝐹 = 2∗ (10)
( ) repository sets”, Elsevier, 2014, pp. 701-709.
(e) Accuracy: Classification accuracy is defined as [7] Lale Ozbakır, Adil Baykasoglu, Sinem Kulluk, “A soft computing-
"percentage of correct predictions". based approach for integrated training and rule extraction from
( ) artificial neural networks: DIFACONN-miner”, Applied Soft
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = (11) Computing 10 (2010), Elsevier,pp. 304–317
( )
[8] Marconi de Arruda Pereira, Clodoveu Augusto Davis Júnior, Eduardo
Gontijo Carrano, João Antôniode Vasconcelos,“A niching genetic
Various parameters that are used in the classification method programming-based multi-objective algorithm for hybrid data
is shown in Table IV. classification”, Elsevier, 2014, pp. 342-357
[9] Mete Celik, Dervis Karaboga & Fehim Koylu, “Artificial Bee Colony
Data Miner (ABC-Miner)”, IEEE, 2011, pp. 96-100
TABLE IV: PARAMETER SETTINGS [10] Pavel Kromer, Jan Platos, Vaclav Snasel, Ajith Abraham,“Fuzzy
Parameters\Method SVM PSO SSO CS Classification by Evolutionary Algorithms”,IEEE,2011,pp.313-318.
Gamma 0.5 [11] Soumadip Ghosh, Sushanta Biswas , Debasree Sarkar ,Partha Pratim
Nu 0.5 Sarkar,“A novel Neuro-fuzzy classification technique for data
Swarm Size 20 mining”, Elsevier, 2014, pp. 1-19.
[12] U. Fayyad, G. Piatesky-Shapiro, P. Smyth, "From Data Mining to
C1,C2 1.5,1.7
Knowledge Discovery in Databases", AI Magzine, vol. 17, no. 3, pp.
Nests 25 37-54, 1996
Pa 0.25 [13] Xiaoyong Liu1, and Hui Fu, “PSO-Based Support Vector Machine
Cg,Cp,Cw 0.55,0.75,0.95 with Cuckoo Search Technique for Clinical Disease Diagnoses”,
Nsol,Ngen,Nrun 150,100.30 Hindawi Publishing Corporation, 2014, pp.1-7
[14] Xin-She Yang, Suash Deb, "Cuckoo Search via Levy Flights", IEEE,
pp. 210-214, 2009.
TABLE V: COMPARISON OF RESULTS [15] Yao Liu, Yuk Ying Chung, Wei Chang Yeh, "Simplified Swarm
Name Existing Approach(CS- Proposed Approach Optimization with Sorted Local Search for Golf Data Classification",
PSO-SVM) (SSO-PSO-SVM) IEEE World Congress on Computational Intelligence, 2012.
Acc Rec Pr F1 Acc Rec Pr F1 [16] Ziqiang Wang, Xia Sun, and Dexian Zhang,“A PSO-Based
Cancer 73 100 73 84.3 100 100 100 100 Classification Rule Mining Algorithm”, Springer, 2007, pp. 377-384.
[17] UCI Machine Learning Repository, http://archive.ics.uci.edu/ml