Extracting Rules from Fuzzy Neural Network by Particle Swarm Optimisation

He Zhenya, Wei Chengjian, Yang Luxi, Gao xiqi and Yao susu Southeast University, Dept. o f Radio Ensneering Nanjing 210096, P.R. China Russell C. Eberhart and Yuhui Shi Purdue School o f Engineering and Technology 799 W. Michigan St., ET215 Indianupolis, IN 46202

Abstract
A four-layer fuzzy neural network is presented to realise
knowledge acquisition from input-output samples. The network parameters including the necessary membership functions of the input variables artd the consequent parameters are tuned and identiyied using a modiyied particle swarm algorithm which uses each particles best current performance of its neighbows to replace the besf prwious one and uses a not accumulative rate o f change to replace the accumulative one for accelerating search procedure. The trained network i s then pruned so that the general rules can be extracted and explained. The experimental results have shown that the similar classification rules can be obtained in comparison to that o f otherfuzzy neural approaches.

algorithm for training and pruning the network The conventional training algorithms are gradient decent method and evolutionary computing method The goal of network pruning is to make a fat network slim and make the network manageable for knowledge extracting task. The third phase is to acquire knowledge and explain rules after pruning the network. In this paper, we propose a new method for extracting knowledge using an adaptive fuzzy neural network. A modified version of particle swarm optimisation is apglied to train the fuzzy neural network, which uses each particles best current performance of its neighbows to replace the best previous one and uses a not accumulative rate of change to replace the accumulative one to accelerate search procedure. The @cIe swarm algorithm like evolutionary computing methods searches for the optimal solutions from multiple directions but requires only primitive mathematical operators and is computationally inexpensive in terms of both memory requirements and speed. The network is then pruned by selecting stronger finng values of the consequent parameters. Finally, the proposed method was examined for the bench mark example classification of Iris species. The knowledge of classification was succesmly extracted. In the following several sections, the f u z z y neural network architecture, particle swarm learning algorithm and experimental results are described in details.
2. Fuzzy NeuraI Network

1. Introduction
Extracting general rules from input -output samples has being an interesting problem in many research fields, such as classification and pattern recognition. Over the last few years, a number of methods of extracting symbolic rules Using artificial neural network methods have been reported. Yoon et a1 proposed a rule extraction method using neural network by destructive learning [11. Hayashi et ai tried to extract fuzzy rules of k conjuncts [2]. Horikawa et ai [3] presented a fuzzy modelling method based on if-then rules using gradient descent learning. Feldman [4], Karr [SI used genetic algorithm for generating if-then rule and tuning the membership functions of antecedent and consequent fuzzy sets. Generally, extracting knowledge based on artificial neural network or fuzzy neural network consists of three main phases. One is to design an adequate network structure that may contain possible configurations of all rules and easy to be trained. Due to fuzzy systems are designed to operate w i t h linguistic variables, the combination of fuzzy systems and neural networks has shown their effectiveness in knowledge acquisition. Another phase is to use an

Structure and Particle Swarm Learning Algorithm

1) Fuzzy neural network architecture Fuzzy logic provides an mference morphology that is associated w i t h human cognitive processes, such as thinlcing and reasoning. On the other hand. neural networks offer exciting advantages such as learning, adaptation, fadtolerance, parallelism and generalisation. To enable a system to deal with cognitive uncertainties in

0-7803-4869-9/98 $10.00@1998 EEE

74

a manner more like humans, a better way is to incorporate the concept of fuzzy logic into the neural network to construct fuzzy network. Several authors have studied fuzzy neural network structures and learning algorithms [6][7].Jang [8] proposed an adaptive-network-based fuzzy inference system, which serves as a basis for generating a set of fuzzy if-then rules. Horikawa et a i presented three types of fuzzy neural network with sigmoid active functions [3]. These networks realise fuzzy implication and reasoning, which are based on Takagi and Sugenos method [9].They proposed a mathematical tool to build a fuzzy model of a system, which can be eqressed by

rules.

2) Particle Swarm Learning Algorithm Particle swarm algorithm is an optimisatilon paradigm

that was discovered through simulation of a simplified social model by Eberhart and Kennedy in 19195 [ 101[111[121[131. The algorithm models the exuploration of a problem space by a population of individuals, individuals successes influence their searches and those of their peers. The original version of the algorithm is as follows: a potential solution to a problem is represented as a particle having coordinates xld and rate of change vfd in a hyperspace. Each particle i maintains a record of the position of its previous best performance in i2 vector called pld. An iteration comprises evaluation of each particle, then stochastic adjustment of Vld in the direction of particle is best previous position and the best previous performance of its neighburs. Neighbourhood can be defined in Werent ways. The variable g is assigned the value of the index of the particle with the best previous performance in the neighburhood. The steps are: 1. Initialise an array of particles with random positions and velocities on D dimensions, 2. Evaluate the desired minimisation hiction in D variables. 3. Compare evaluation with particles prevous best value (PBESTU): If current value <PBEST[] then PBESTU=currentvalue and PBESTx[][d]= current position in D dimensional hyperspace. 4. Compare evaluation with groups previous best(PBEST[GBESTl):If current value < PBEST[GBEST] then GBEST=particlesarray index. 5. Change velocity by the following formula: Vu[d]= V[][d] + ACC-CONST * rand0 * (PBESTxn [d] - Presentxu [d]) + ACC-CONST * rand() * (PBESTx[GBEST][d]-PresentXU [d]), 6. Move to PresentXl][d]+V[][d]; Loop to step 2 and repeat until a criterion is met. In this paper we m&ed the steps 4 and 5 of the original particle swarm version by 4 and 5 as follows:
4. Find each particles current best (PresentX[GBEST])in its neighbours: If current value < PresentX[GBEST] then GBEST=particlesarray index, 5. Change velocity by the following formula:

where, X, ; . -, Xt are input variables. A, 7 - A, denote fuzzy sets with appropriate membership functions, which are also referred to as premise parameters. po;-, p k represent parameters of consequence. This fuzzy model has been successfully used for fuzzy identification and non-linear modelling. For classification, more than one output node is needed depending upon the number of objects to be classified. A simple case is only to idenw po of the consequent part. Based on the above fuzzy system the corresponding fuzzy neural network can be constructed Fig.1 gives an architecture of fuzzy network for classification. For simplicity, only the case of two input variables and two output nodes is shown. This fuzzy network is composed of four layers containing fuzzification of the input variables, fuzzy rule operation and defuzzification functions.

Layer 1 calculates the value of the membershp Eunction that is of bell-shaped:

A(x) =

where x is the input variable and a, b, ,Cl are the premise parameters. The shape and position of the membership function will change with any of a, , b, or C ,. Layer 2 performs and operation of incoming signals. It is equivalent to the meaning of the firing strength in fuzzy system.
to the sum of all the

Layer 3 calculates the i-th firing strength proportional firing strengths.

PresentX[J[d])+ ACC-CONST * rand0 * (PresentX[GBESTl[d] PresentX[I[dl).

V[][d]=ACC-CONST*rand()*(PBESTx[][d]-

Layer 4 is the output layer, the number of nodes correspnds to that of objects to be classified Each node in this layer sums all the action values from all the inference

and got a better version of particle swarm search. The following two test functions are used to demonstrate the effectiveness of the modified version:

75

The first function is:

f u z z y neural network has four input nodes and three

output nodes. Two membership functions are used for each input variable. The data set is divided into two groups of training and test sets. Each group contains 75 seven dimensional vectors. The fuzzy neural network is first trained with the training data set and then checked using test set. When the network is trained 600 epochs and 15 rules are extracted to describe the data species a 97% classfication rate is achieved However, the classification error is increased when fewer rules are used. Three strongest rules are successfully e"acted whxh are similar to the results reported in [151:

F(x, y ) = x 2 + 2y2 - 0.3 cos(3m)0.4 cos(4ny) + 0.7.

It has numerous local minima and a global minimum of zero at the origin. The second function is Rosenbrock function: F(x,y) = 10O(x2- y y 4 (1- x)2., It has a parabolic valley along the curve y=x2 , w i t ha unique minimum of zero at the point (1,l). There are no local optima associated with function. The Fig. 2 and Fig. 3 show the diference of the o r i p a l particle swarm search (dashed line) and the modified one with x,y ranging over [-30, 301. When applying particle swarm algorithm to optimise fuzzy neural network, the 20 initial particles that represent the premise and consequent parameters to be tuned are initialised randomly, parameter having components valued between -0.5 and +OS ( premise parameters are assigned according to fuzzy linguistic values). The maximum velocity of the particle swarm is limited to 0.35 and the acceleration constant is specified to 2.0.

IF high2 AND low3 AND low4 THEN class 1 IF high2 AND high3 AND low4 THEN class2 IF high2 AND high3 AND high4 THEN class3

5. Conclusions
We have described the architecture of fuzzy neural network for classification. The fuzzy implication and reasoning are realised using the proposed fuzzy network. To overcome the drawbacks of gradient information and evolutionary computing based learning algorithm, we apply a modified version of particle swarm algorithm to tune the premise parameters and consequent parameters. The m o d i f i e d version of particle swarm algorithm uses each particle's best current performance of its neighburs to replace the best previous one and uses a not acmulative rate of change to replace the accumulative one which accelerate the seasch procedure.
~

3. Knowledge Acquisition
After training fuzzy neural network, the knowledge is permeated into its own particular knowledge structure. That is, knowledge is contained in network topology and parameter sets. In general, this knowledge is very diacult to interpret because there exist redundant rules or rules overlapping. Before e'xtracting rules from the trained network we have to prune redudant components fiom the network to make the network manageable for the d e extracting goal. We can take advantage of a small network and extract a relatively small d e set. The strategy of pruning network used in this work is very simple. That is, for all the remaining links in the layer four, set those weaker firing strengths under a certain threshold zero and retrain the network After replacing the firing strength, the network error will be increased. Therefore the all the parameters of the fuzzy neural network including premise and consequent parameter sets need to be retrained using particle swam learning algorithm. The firing strength for each rule is distributed again among the rule links. This process repeats until a relatively small rule set and minima network error are achieved.

References
1.

B. Yoon and R.C. Lacer, Extracting rules by destructive

learning, The 1994 Intemational Conference on Neural Networks, Ireland, Florida, pp. 1176-1771. 2. Y. Hayashi, et ai., Hybrid expert system with fuzzy ifthen rule generation, Proceedings of the Intemational Joint ConferenceJ 994.on Neural Networks, Beijing, China,pp.m:liO 1-506. 3. S. Honlcawa, et al., On fuzzy modelling using fuzzy neural networks w i t h the back-propagation algorithm, IEEE Trans. Neural Networks, 3(5), pp.801-806, 1992. 4. D.S. Feldman, Fuzzy network synthesis with genetic t h ICGA, University of Illinois at algorithm, Proc. 5 Urbana Champaign 312-317,1993. 5. C.L. K m , Design of an adaptive fizzy logic controller uskg a genetic algorithm, Proc. 4th ICGA, San Diego,
Ca, USA, 450457,199 1.

6.

7.

4. Application to the Iris Classification

The proposed method is applied to a benchmark example known as classification of Ins species [14]. The data contain a set of four dimensional vectors. each of which represents sepal length. sepal width, petal length and petal width of one of three Ins subspecies Setosa, Versicolor and Virginica. Considering the Iris data type, the proposed 76
8.
9.

J.J. Buckley and Y. Hayashi, Fuzzy neural nets and applications, Fuzzy Systems and AI, 3, pp. 11-11, 1992. L.X. Wang and J.M. Mendel, Fuzzy basis fimctions, universal approximation and orthogonal least squares learning,IEEE Trans. Neural Networks, 3(5), pp.807-8 14, 1992. R. Jang, ANFIS: Adaptive-network-based fuzzy inference system, IEEE Trans. on Systems, Man and Cybernetics, Vol. 23, pp.665-683, 1993. T. Takagi and M. Sugeno, Fuzzy identification of systems and its applications to modeling and control, IEEE Trans.

on System, Man, and Cybernetics, Vol. SMC-15, No.1, 1985. 10. R.C. Eberhart, A new optimizer using particle swarm theory, Sixth International Symposium on Micro Machine and Human Science, 39-43. 11. J. Kennedy and R.C. E k M , Particle s w a optimisation, IEEE Conference on Neural networks, 1995, A d a . 12. J. Kennedy, The particle swarm: social adaptation knowledge, ICEC97 303-308.
p ~ 116-132, .

13. J. Kennedy and R.C. Eberhart, A discrete binary version of the particle swarm a l g o r i h IEEE Conhence on System, Man, Cybernetics, 1997. 14. Ehderson, The irises of the gaspe ~leninsula, Bill. Pamer. Iris Soc.59(1935) 2-5. 15. S.K. Haigamuge, et al, A new method for generating fuzzy classification systems using rbf neurons w i t h extended rce learning, The 1994 Intematio,nalConference on Neural Networks, Orlando, Florida, pp. 1589-1594.

Y,

Y 2

Fig. 1. The fuzzy neural network for classification

E
L

1os

.- - I

---0
100 200 300

.
I

I .

m
iterations

Q)

400

IU

100

200
Iterations

300

400

Fig. 2 The first function search procedure

Fig. 3 The second function search procedure

77