AI TERM PROJECT GROUP 14

Anti-Spam Filter Based on Nave Bayes, SVM, and KNN model

Yun-Nung Chen, Che-An Lu, Chao-Yu Huang
AbstractNave Bayes spam email filters are a well-known and powerful type of filters. We construct different filters using three types of classification, including Nave Bayes, SVM, and KNN. We compare the pros and cons between these three types and use some approaches to improve them to get a better spam filter. Index TermsSpam filter, Nave Bayes, SVM, KNN

1 INTRODUCTION
ass unsolicited electrontic mail, often known as spam, has recently increased enormously and has become a serious threat to not only the Internet but also to society. Over the past few years, many approaches have been provided to filter spam. Some of them often use Nave Bayes method to implement it.

2 PROPOSED APPROACHES
2.1 Problem Definition Construct a filter which can prevent spam from getting into mailbox. Input: A mail message Output: Spam or Ham 2.2 Proposed Solution

Nave Bayes, SVM, and KNN. We will compare these three methods first. After that we will modify each method independently to improve the accuracy and compare each method itself. Finally make a conclusion above all the improvement on these three methods. Following are our methods details. Nave Bayes From Bayesian theorem of total probability, given the vector x = x , x , , x of a mail d, the probability that d belongs to category C is:

| = Pc = C| X x =
where k spam, ham.

(1)

Figure.1 System Flowchart We use method of machine learning to train a model, and decide the input message belonges ham or spam. We use three methods to implement spam filter, including

Yun-Nung Chen Author is in the National Taiwan Universty. E-mail: swallow29271223@gmail.com. Che-An Lu Author is in the National Taiwan Universty. E-mail: JAST.Lu@gmail.com. Chau-Yu Huang Author is in the National Taiwan Universty. E-mail: hamigwa@gmail.com

We can decide that mail belongs to a model with a higher probability. This uses unigram language model to compute the probability of class. In order to get better feature (word with more information), we preprocess testing message to remove some noise word and reserve words which are more important. We remove words whose lenth is longer than 50, and we also delete words which appear for less 20 times. A word which appears in more documents represents that it carries less information for classification. We use SRI Language Model Toolkit: to generate unigram language model, and compute the Bayesian Probability according to the model. We can decide the classification of testing message. SVM The support vector machines (SVM) is easy to use and powerful for data classification. When generating filter model, we create a vector for each data in tha training corpus and then SVM will map these vectors into hyperplan. Then SVM will learn which kinds of vector are close to which class. Here SVM is a good approaching method in precisely finding best classification hyper-plans to maximize the margin so that we could classify a new mail into spam or ham.

2009 AI TERMPROJECT

AI TERM PROJECT GROUP 14

cost and gamma to the best condition. KNN Version1 First we will create a binary vector which maintain the information of each feature (term) exist or not (1 or 0) for each training data. Secondly, for each unclassified mail we will create a binary vector too, then using cosine distance to find out the top K closest training data from the corpus then find out which class is the unclassified mail belongs to.

cosine distance:

Figure.2 Version1 First we will select top 1000 document frequency terms as we do in KNN. Secondly we will create TF-IDF for each training data. Thirdly we will use libsvm (a tool from Chih-Jen Lin) to train a model. With each unclassified mail we will create a TF-IDF vector too, finally predict it with the svm model we trained before. The fomula of TF-IDFis described as below:

Version2 It is similar to version 1, one big difference is that in version 2, we will not use binary vector as before, we will use TF-IDF respectively. Version3 It is similar to previous version, we only turn the uppercase into lowercase (case-insensitive), so that for example free will be see as the same term with FREE or Free.

| |

(3)

0.5 + 0.5 log + 1

3 CONTRIBUTIONS
(2)

Where N is number of terms in whole corpus, f is frequency of term t, and df is data frequency of term t. If a term appears in many documents, it represents that this term doesnt have significance. So we use IDF to diminish the weight of such term inorder to select better features. Why we use data frequency to select the feature is because the following figure (a reference from course Web Mining).

Figure.3 We can see that the performance between all the feature selection methods except mutual information and term strength makes no much difference. For our coding convinence, we choose to implement data frequency. Version2 It is similar to version 1, we only turn the uppercase into lowercase (case-insensitive), so that for example free will be the same term with FREE or Free Version3 It is similar to version 2, the only difference is that when training svm model, we will set the parameters

3.1 Compare each method independently We compare three methods independently, and we also can observe the difference between sizes in 1200, 3000, 9000, and 21000. Nave Bayes When we trains language model, computing the probabilities of words doesnt care the case (case-insensitive), and we also remove the word with too small probability for message preprocessing. When the size of traing data is smaller (1200), the result still has good performance. We can see that accuracy is 93.6206%. But when training set becomes a little larger (3000), the result is not good as smaller one. We can see that the result is improved up slightly. Compared to size, the improvement of accuracy is relatively small, and we can just improve accuracy to 3% (93.6206% -> 96.2669). We believe feature selection is important to Nave Bayes, and we can use a better feature to improve result. But we must spend much time to testing data to see whether the result will be improved up, time and accuracy is tradeoff. SVM When the training set is small (1200), the result of SVM model is much poor than others. When we use case-insensitive to create tf-idf vector, the accuracy can improve up to 20% (60.854% -> 72.6285%), which means that it is important to combine the information of uppercase and lowercase together to increase the concept for a specific term (ex: free, Free, FREE). The other reason is that if we see free and Free as the same term, then the data frequency of free will increase, so that we wont throw away such important feature (since we only select top 1000 ranked by data frequency).

AI TERM PROJECT GROUP 14

After we find the best gamma and cost value for each corpus, the performance can also improve up to 20% (72.6285% -> 88.2511%). But the process of finding such parameters is very time consuming, so it will be a tradeoff. When using large training set, the performance wont improve accordingly. We think that it is because when the training set grows up, the noise will also increase, which means that there will be more ham similar to spam. KNN The performance of KNN is really out of our expectation. At begin, we think KNN wont be better than SVM. But after our experiment, KNN seems work well on spam classification. The other interesting thing is that when using binary vector and TF-IDF vector, the performance makes little difference (notice that TF-IDF still a little better than binary vector). The improvement of using TF-IDF vector is not that significant as we expect. We think that it is because in spam classification, some important features in spam is quite different from ham, so the weight (TFIDF) of such feature will not be so important as whether there exist such feature. And we also found that the case-sensitive and caseinsensitive makes little difference in KNN, not like in SVM. We think the main reason is that we didnt do feature selection in KNN, instead, we keep all the features. So we wont throw out some important features (ex: Money) such as SVM feature selection does. And this may also be the reason why KNN will beat SVM.

The testing data are 2910 ham and 4912 spam which are randomly select from the corpus. Remaining data (10000 ham, 20000 spam) are used to be training data. These two set (testing data and training data) is independent. In our experiments, we will create four different sizes of training data which are randomly select from the training corpus. The ratio of spam and ham in these four training datas are all 2:1, and the corresponding size of them are 1200, 3000, 9000, and 21000. (Following we will use 800:400, 2000:1000, 6000:3000, 14000:7000 to represent four different training sets)

3.2 Compare all these three methods We think that it is why KNN is better than Nave Bayes that the feature we select when we implement Nave Bayes is not good enough to train an excellent model. We think KNN can beat SVM is because that we throw some information when doing feature selection in SVM. And the other reason might be that spam classification problem is a binary decide problem (spam / ham), so KNN can easily close to one side, and we think if there exists more class, the performance of SVM will be better than KNN. When training corpus is small, the performance of KNN is still well, not like SVM, with only 60% accuracy. We think it is because SVM is a machine learning method, we cant expect it learns well with a little training data. KNN can still find the top K similar data in a small corpus. We also think that Nave Bayes is good method to filt spam when traing set is small, and because the word appearing in ham isnt too many, we can compute probabilities of words to decide a category by using a training set with small size.

4.2 Result of Evaluation Following are three different methods accuracy table with different sizes of training data. Nave Bayes Accuracy 800:400 93.6206 % 2000:1000 90.8719 % 6000:3000 95.2570 % 14000:7000 96.2669 % Table. 1 SVM Version1 Version2 Version3 800:400 60.8540 % 72.6285 % 88.2511 % 2000:1000 87.4585 % 90.5316 % 93.8794 % 6000:3000 88.8520 % 93.5983 % 96.4477 % 14000:7000 88.2255 % 91.0299 % 91.0299 % Table. 2 KNN Version1 Version2 Version3 800:400 94.0710 % 95.6172 % 95.6044 % 2000:1000 95.9877 % 97.2144 % 97.0611 % 6000:3000 97.5466 % 98.0833 % 97.7383 % 14000:7000 97.1122 % 97.8789 % 97.5978 % Table. 3 Fig.4 is the accuracy plot of Nave Bayes, first version of SVM, and first version of KNN with different size of training data.
100.00% 95.00% 90.00% 85.00% 80.00% ver1 SVM 75.00% ver1 KNN 70.00% 65.00% 60.00% 800:400 2000:1000 6000:3000 14000:7000 Nave Bayes

4 EXPERIMENTAL RESULTS
4.1 Corpus We use the corpus which is provided by trec06. There are 37822 messages (12910 ham, 24912 spam). We separate the whole set into training data and testing data.

Figure.4 Accuracy of Nave Bayes We use Nave Bayes to be the baselin. We can see that in the first version of SVM, all the accuracy are less than Nave Bayes. And in the first version of KNN, the performance is already better than Nave bayes. Fig.5 is the accuracy plot after we improve the SVM

AI TERM PROJECT GROUP 14

accuracy.
100.00% 95.00% 90.00% 85.00% ver1 SVM 80.00% ver2 SVM 75.00% 70.00% 65.00% 60.00% 800:400 2000:1000 6000:3000 14000:7000 ver3 SVM

100.00% 98.00% 96.00% 94.00% 92.00% 90.00% 88.00% 86.00% 800:400 2000:1000 6000:3000 14000:7000 Nave Bayes ver3 SVM ver2 KNN

Figure.7 Comparison of 3 methods Figure.5 Accuracy of SVM We can see that there has a big improvement between version 1 and version 2, the difference between these two versions is that we use case-insensitive in version 2. In version 2 we will much emphasize some important features such as Free, and we wont throw too many information away due to feature selection as we mentioned before. The improvement between version 2 and version 3 is also significant. By well selecting the SVM parameter gamma and cost, the performance can really improve a lot as we can see. Fig.6 is the accuracy plot after we improve the KNN accuracy.
100.00% 99.00% 98.00% 97.00% 96.00% 95.00% 94.00% 93.00% 800:400 2000:1000 6000:3000 14000:7000

As we can see, after we improve SVM, two of the middle size of training set is better than Nave Bayes. And the other two is much more close to Nave bayes compare with the first version. After we improve KNN, the result is again better than Nave Bayes.

5 CONCLUSION
After experimenting three different methods, we found that KNN has higher accuracy than other two approaches. Because we think KNN is more suitable for classification of less catergories than SVM, accuracy of KNN is higher. We think Nave Bayes is a good method for spam filter, and the time costs little on training (about 1-2 seconds). Testing an input message requires much time using Nave Bayes, but the result is good enough. The training time of KNN is very fast, but it takes lots of time on testing. We think it is because we didn implement any indexing algorithm such as KD-tree, R-Tree or Quad-Tree when finding the nearest top K neighbors. In our future work, we can implement these indexing methods to improve the efficiency of KNN. The training time of SVM compared with Nave Bayes and KNN is much longer, espically when we want to find out the best gamma and cost parameters for the training process. But the testing time of SVM is much faster than the other two methods. In future work, we can focus on different feature selection methods to improve the performance of Nave Bayes and SVM, and the results of them might become better than KNN.

ver1 KNN ver2 KNN ver3 KNN

Figure.6 Accuracy of KNN We can see that there is a little improvement from version 1 to version 2, the difference of them is that we use TF-IDF vector instead of binary vector in version 2. And when we modified version 2 from case-sensitive to case-insensitive, the difference between them is not as significant as SVM (no more than 0.4%). We also mentioned this before, it is because we didnt do feature selection in KNN. Fig.7 is the accuracy plot of Nave Bayes, and the best version of SVM and KNN with different size of training data.

JOB RESPONSIBILITY

Yun-nung Chen (B94902032) Nave Bayes (Training and Testing), Report Writing. Che-an Lu (B94902097) SVM (Training and Testing), KNN(Training and Testing), Report Writing. Chau-yu Huang (B94902052) Message Preprocessing, Report Writing.

ACKNOWLEDGMENT
The report uses lots of toolkits, including MIME-tools,

AI TERM PROJECT GROUP 14

SRILM, and SVM. So we want to thanks about it.

REFERENCES
[1] [2] SRI Language Model Toolkit http://www.speech.sri.com/projects/srilm/ CJ Lins Home Page http://www.csie.ntu.edu.tw/~cjlin/