Enhanced SMOTE Algorithm for Classification of
Imbalanced Big-Data using Random Forest
Reshma C. Bhagat
M. Tech. Student Department of CSE
Rajarambapu Institute of Technology
Islampur (Sangli), MS, India
resham.bhagat3@gmail.com
Abstract— In the era of big data, the applications
generating tremendous amount of data are becoming the
main focus of attention as the wide increment of data
generation and storage that has taken place in the last few
years. This scenario is challenging for data mining
techniques which are not arrogated to the new space and
time requirements. In many of the real world applications,
classification of imbalanced data-sets is the point of
attraction. Most of the classification methods focused on
two-class imbalanced problem. So, it is necessary to solve
multi-class imbalanced problem, which exist in real-world
domains.
In the proposed work, we introduced a methodology for
classification of multi-class imbalanced data. This
methodology consists of two steps: In first step we used
Binarization techniques (OVA and OVO) for decomposing
original dataset into subsets of binary classes. In second
step, the SMOTE algorithm is applied against each subset
of imbalanced binary class in order to get balanced data.
Finally, to achieve classification goal Random Forest (RF)
classifier is used. Specifically, oversampling technique is
adapted to big data using MapReduce so that this
technique is able to handle as large data-set as needed.
An experimental study is carried out to evaluate the
performance of proposed method. For experimental
analysis, we have used different datasets from UCI
repository and the proposed system is implemented on
Apache Hadoop and Apache Spark platform. The results
obtained shows that proposed method outperforms over
other methods.
Keywords— Data mining; Multi-class Imbalanced data;
Oversampling; MapReduce; Machine Learning.
I. INTRODUCTION
The appearance of information technology in various filed
of our life lead to the massive amount of data storage in
various formats like records, documents, images, sound
recordings, and scientific data and so on. Now-a-days the data
is collected from various domains and it requires well defined
methods for extracting knowledge or information from this
data for better decision making. Knowledge discovery in
database often called as Data Mining. Data mining came into
existence due to the perception of ʊwe are data rich but
information poor.
978-1-4799-8047-5/15/$31.00 2015
c IEEE
Sachin S. Patil
Assistant Professor, Department of CSE
Rajarambapu Institute of Technology
Islampur (Sangli), MS, India
sachin.patil@ritindia.edu
A. Basics of Data Mining
Data mining is the extraction of predictive information
from large data-sets. The various data mining tools are used to
predict useful information from available data-sets, helps
organizations to make proactive business-driven decisions.
Various data mining techniques are present including
association rules, decision tree classification, clustering, and
sequence mining and so on. Classification is most popular
application area of data miming. Classification is a process of
assigning new objects to predefined categories or classes. The
classification plays an important role and is essential tool for
several organizations and in many individual’s lives.
Classification come into existence to makes our life better and
easier. [4] Machine learning is defined as ʊalgorithmic part
of data mining process. During the last few years, machine
learning techniques have been applied to complex real-world
problems with the aim of extracting novel and useful
knowledge [4] [16].
Furthermore, many real world applications present classes
which have insignificant number of samples as compared to
other class that are considered. This situation is called as class
imbalance problem. Usually, the insignificant samples are
main focus of study, so it is necessary to classify them
correctly. In machine learning research, learning from
imbalanced data-sets is an issue that has attracted a lot of
attention. The statistical learning methods are suited for
balanced data sets and may ignore the negligible samples
which are important. That‘s why it is required to consider the
features of problem and solve it correctly.
Another challenge for data mining algorithms is big data.
The techniques which are used to deal with big data are
concentrated on gaining fast, scalable and parallel
implementations. To reach this goal, MapReduce framework
is followed. MapReduce framework is one of the most popular
procedures used for handling big data. The working of this
framework based on ʊdivide and conquers strategy, where
the dataset is divided into subsets which are easy to manage
and partial results are combined that are obtained.
B. Big Data and Hadoop
Big data is data that overshoot the processing capacity of
traditional database systems. The data is too large, moves too
fast or doesn‘t fit into predefined structure of database
architecture. From 2012 Big Data has become a hot IT
403
buzzword and it is defined in terms of 3 V‘s such as Volume,
Velocity and Variety [1] [19].
In 2004, the MapReduce programming framework was
first proposed by Google. It is a platform designed for
processing tremendous amounts of data in an extremely
parallel manner. It provides an environment to easily develop
scalable and fault tolerant applications. The Apache Hadoop is
an open source project and provides the MapReduce
framework for processing big data [1].
Apache Mahout is an open source project and runs on top
of Hadoop platform. It provides set of machine learning
algorithms for clustering, recommendation and classification
problems. Mahout contains various implementations of
classification models such as Logistic Regression, Bayesian
models, Support Vector Machines, and Random Forest among
others [2].
Like Hadoop, Apache Spark is an open-source cluster
computing framework. It is developed in the AMPLab at UC
Berkeley. It is in-memory based MapReduce paradigm. It
performs 100 times faster than Hadoop for certain applications
[3].
In this work, we present analysis of techniques which are
used to deal with imbalanced data problem. We present the
enhanced SMOTE algorithm for handling multi-class
imbalanced big data problem. These approaches are evaluated
on the basis of potency in correct classification of each
instance of each class and time required to build classification
model. In order to perform classification, Random Forest
classifier (RF) which is a popular and well-known decision
tree ensemble method is used. It is proven that RF is scalable,
robust and gives good performance.
For experimental study, we will focus on MapReduce
based implementation of SMOTE + RF. The experiments
performed provide the limitations of original multi-
classification algorithm and enhanced SMOTE algorithm.
Finally, we evaluate the proposed system based on accuracy
and Geometric Mean for true rates and ȕ-F-Measure - popular
measures in imbalanced domain.
II. RELATED WORK
A. Imbalanced Data Problem
The classification of Imbalanced datasets poses problem
where class distributions have number of examples in one
class is outnumbered by the number of examples in the other
class.[5][6] A class having abundant number of examples
called majority or negative class and a class having significant
number of examples called minority or positive class. In
recent years, the imbalanced data problem has become
burning point in industry, academia and government agencies.
This problem present in many real world problems such as
medical diagnosis [8], fraud detection, finances, risk
management, network intrusion, E-mail foldering [12],
Software Defect Detection [18] and so on. Additionally, the
positive (minority) class is the class of interest from the
learning point of view and has great impact when it is not
classified properly.
404 2015 IEEE International Advance Computing Conference (IACC)
B. Addressing Imbalanced Problem
Several techniques are present to address the classification of
Imbalanced data [5] [6] [19]. These techniques are categorized
into various groups:
1. Data Level Approach: In which original dataset is
modified to get balanced dataset so that it can be
used by standard machine learning algorithms.
2. Algorithm Level Approach: In which existing
algorithm is modified to launch procedures that can
deal with imbalanced data.
3. Cost-sensitive Approach: In which both data level
and algorithm level approaches are combining to get
accuracy and reduce misclassification costs.
Furthermore, data level approaches are divided into
various groups: Oversampling technique and Undersamping
technique and hybrid technique. In oversampling technique,
new data from minority classes are added to original dataset in
order to obtain balanced dataset. In undersampling technique,
data from majority classes are removed in order to balance the
datasets. In hybrid technique, previous techniques are
combines to achieve the goal of balanced dataset. Usually, at
first oversampling technique is used to create new samples for
minority class and then undersampling technique is applied to
delete samples from majority class [5] [6] [11].
The oversampling and undersampling techniques have
some drawbacks. To address this end, Synthetic Minority
Oversampling Technique (SMOTE) is used. The SMOTE
algorithm is a powerful solution to imbalance data problem
that have shown success in various application domains. The
SMOTE algorithm is an oversampling technique. This
technique adds synthetic minority class samples to original
dataset to achieve balanced dataset [5] [6].
In SMOTE algorithm, minority class is oversampled by
duplicating samples from minority class. Depending on the
oversampling required, numbers of nearest neighbors are
randomly chosen. The synthetic data is generated based on
feature space likeliness prevails between existing samples of
minority class. For subset ‫ א‬S, consider K nearest neighbors
for each sample ‫ א‬X. The K-nearest neighbors are the K
elements whose Euclidean distance between itself and have
smallest weight along the n-dimensional feature space of X.
The samples are generated simply as, randomly select one of
the K-nearest neighbors and multiply corresponding Euclidean
distance with random number between [0,1]. Finally, add this
value to original value of instance [5].
Mathematically, it is written as
Where, Ǧ ample from minority class used to generate
synthetic data. - Nearest neighbor for and į - Random
number between [0, 1]. The generated synthetic data is a point
on line fragment between under consideration and k-nearest
neighbors for Ǥ
Though SMOTE is popular technique in imbalanced
domain, it has some drawbacks including over-generalization,
only applicable for binary class problem, over-sampling rate
vary with dataset etc. To avoid these drawbacks some
approaches are defined such as Borderline-SMOTE and
Adaptive Synthetic Sampling for overgeneralization.
Evolutionary algorithms and sampling methods are used to
deal with class imbalanced problem. [12]
SMOTE+GLMBoost [13] and NRBoundary-SMOTE based
on neighborhood Rough Set Model [15] are used to solve class
imbalance problem. The ensemble methods like AdaBoost,
RUSBoost, and SMOTEBoost coupled with SMOTE to solve
imbalanced data problem [8] [11].
All these approaches are focused on two-class problem. In
[10], author proposed a solution for multi-class problem based
on fuzzy rule classification. In this paper, one-vs.-one
binarization technique is used to transform multi-class data
into two-class data.
III. SYSTEM ARCHITECTURE
In this section, architecture of proposed system is
explained. Fig. 1 demonstrates the working of proposed
system. SMOTE algorithm + one-vs.-all technique used to
balance the imbalanced data. Random Forest is used for
classification purpose.
Fig. 1: Proposed System Architecture
A. SMOTE Algorithm for Big- Data
The SMOTE algorithm will be implementing for big data
following a Map Reduce design where each Map process
oversamples the minority class and the Reduce process
randomizes the output generated by each mapper to form the
balanced dataset. Initial step the algorithm performs a
segmentation of the input dataset into independent data blocks;
replicates and transfers them to other machines.
Fig. 2: Flowchart of SMOTE Map Reduce design
Map phase, each map process balances the class
distribution in a mapper‘s partition using the basic SMOTE
algorithm over the available data. Reduce step collects the
output generated by each mapper and randomizes the final
2015 IEEE International Advance Computing Conference (IACC)
data. Final step, the balanced dataset that is generated in the
Reduce process forms the final output that will be the entry
data for the RF-Big Data algorithm [19].
B. One-vs.-All Approach (OVA)
Multi-classes entail additional complexity to data mining
algorithms, since datasets have overlapped class boundaries,
which degrade the performance level. In such cases, we
transform multi-class problem into subset of binary class
problem, which is easy to distinguish, using class binarization
techniques [7] [10] [14].
Here, we have used One-vs.-All or One-against-All
approach, which construct single classifier for each classes of
problem by considering samples of current class are positive
and remaining samples are negative. After performing OVA
technique, SMOTE is applied for each binary class problem
and results of all binary class problems are combined to get
overall result. Then this balanced data is applied to RF
algorithm for classification.
Fig 3: One-vs.-All Binarization Technique for 4-class problem
C. Random Forest for Classification
Ensembles of decision trees (often known as ĴRandom
Forests‘) have been the most successful general-purpose
algorithm in modern times. Random Forest (RF) is an
ensemble method of classification. RF was proposed by Leo
Breiman in 2002 to improve the classification of dataset
having small training dataset and large testing dataset i.e. RF
is suited for large number attributes and small number of
observations [16].
RF is scalable, fast and durable approach for classification of
high-dimensional data and can handle continuous data,
categorical data time-to-event data etc. [4].
D. Random Forest for Classification with Big Data
To handle big data, original RF algorithm needs to be
reshaped so it can fruitfully handle big data. The Apache
Mahout provides Inmem and Partial implementation of RF.
The Mahout operates on the top of Hadoop and uses
MapReduce framework for implementation of machine
learning algorithms. RF algorithm is working in two phases:
the first phase is dedicated to the building of classification
model and second phase is dedicated to prediction of class
405
labels related to dataset usingIV.
already generated model. for corresponding data blocks based on voting scheme. In
Final step, outputs from all mapper are collected to form final
Algorithm 1: SMOTE in MapReduce prediction of classification.
Input: (Key: Value) pair, where Key is the bytes and
Value is the value of an instance. IV. EXPERIMENTAL FRAMEWORK
Output (Key’: Value’) pair, where Key’ is any value and
Value’ is the value of instance. In this section we present details of real-world problems
having multi-class imbalanced data, experimentation and
configuration parameters and then we presents mathematical
SMOTE Algorithm MAP (Key, Value) tests used to compare performance of different approaches.
1: instance = INSTANCE (value) A. Datasets and Parameters
2: instances: add (instance) For experimentation we have used datasets from UCI
3: smote = Filter.usefilter(instances, SMOTE) database repository. Table I summarizes the details of selected
4: smoteinstances= smote.run() data-sets includes no of examples (#EX), number of attributes
5: for i = 0 to smote instances.length - 1 do (#ATTR), Class Distribution and number of classes (#CL).
6: EMIT (key, smoteinstances.get(i))
7: end for TABLE I. CHARACTERISTICS OF DATASET
SMOTE algorithm REDUCE (key, values)

1: while values.hasNext ()
2: instance = INSTANCE REPRESENTATION
(values.getValue())
3: smote instances.add(instance)
4: end while
5: final instances RANDOMIZE (smote instances)
6: for i = 0 to final instances:length - 1do
7: EMIT (null, finalinstances.get(i))
8: end for

In model building phase, training dataset is used for

creation of model using MapReduce framework. It consists of
three steps: Initial, Map and Reduce. In initial step, original
training dataset is divided into separate data blocks and these
blocks are copied across all the nodes. In Map phase, each
Map task works parallel and builds group of forest with
corresponding data block. Finally, in the Final step the output
files by all the mappers are parsed to extract the trees. The
assembly of all the trees forms the forest.
Algorithm 2: SMOTE+OVA
1. Divide training data-set into N binary subsets
considering all classes.
2. For each binary class I, i=1………N
a. Apply SMOTE preprocessing techniques
b. Combine the results of all binary subsets
3. Apply RF-BigData algorithm over the balanced
V. step.
data returned by previous
4. Compute the accuracy, G-Mean and B-f-Measure.
After building of model is completed, classification is
started to predict the class labels related to dataset. Again, this
phase has three stages: Initial, Map and Final. Like first phase,
in Initial step, the segmentation of data is performed and the
resulting data blocks are duplicated and transferred across all
the nodes. In Map step, each Map task performs classification
SMOTE algorithm generates synthetic data based on
Euclidean distance as mentioned earlier. While running
SMOTE on data we have set some parameters which are
1. Nearest Neighbors (N): How many nearest
neighbors to find. (Default value is 5)
2. ClassValueIndex (C): This parameter sets the
class values to examples. (Default value is 0)
3. RandomSeed (K): This parameter sets the number
of random seeds to be selected from Data-set.
(Default value is 5)
4. Percentage (P): This parameter sets the amount of
oversampling. Its value can be 100, 200, and 300
and so on.
B. Evaluation in Multi-class Imbalanced Domain
Evaluation is important factor both in analysis of
classification performance and guidance to model
construction. The quality of classifier is given in the form of
confusion matrix which arranges instances of each class
according to their correct and incorrect identification.
F-Measure:
It provides a simple way to assess the ability of classifier

406 2015 IEEE International Advance Computing Conference (IACC)

to correctly classify active and inactive. It provides a simple TABLE II. COMPARISON OF ACCURACY FOR TESTING AND TRAINING
way to assess the ability of classifier to correctly classify DATASETS WITH BASE METHOD AND PROPOSED METHOD

active and inactive instances by considering sensitivity and Base (OVA) SMOTE+OVA)
specificity into single metric. F-measure gives balanced
Dataset ACCtr ACCts ACCtr ACCts
accuracy and is defined as follows: [5] [19]
Sensitivity (Recall): Landsat 1 0.6534 0.9999 0.7190
It is defined as percentage of positive samples which are Image segment 0.934 0.9339 1 0.9753
correctly classified samples and it is given as:
Lymphography 1 0.9987 1 1
Iris 0.98 0.9943 1 0.9533
Sensitivity= Zoo 1 1 1 1
Car 0.9008 0.9343 0.9867 0.9305
Specificity:
It is defined as percentage of negative samples which Vehicle 0.8156 0.8158 0.9976 0.9976
are correctly classified and it is given as: Waveform 0.8736 0.8697 0.9996 0.9995
Mean 0.9381 0.90 0.9978 0.9469
Specificity=
Precision: TABLE III. COMPARING F-MEASURE VALUE FOR TESTING
DATASET WITH BASE METHOD AND PROPOSED METHOD
It is the proportion of predicted positive cases that were
correctly classified. Dataset BASE(OVA) SMOTE+OVA

Precision= Landsat 0.654 0.702

Image segment 0.924 0.978

For multi-class data, assume K be the number of classes, Lymphography 0.998 1

be the precision of class and be the recall of class. Iris 0.992 0.955
So, G-mean is calculated as: Zoo 1 1
Car 0.882 0.819
G-Mean=
Vehicle 0.752 0.998
F-measure:
Another metric used to assess quality of classifier in Waveform 0.805 0.999
imbalanced domain is -f-measure and it is given as: Mean 0.8758 0.9314

The results obtained shows that both accuracy and F-

F-measure= for all i i=1………K Measure value gives best results in test correspond to one
that is obtained by SMOTE+OVA method.

TABLE IV. COMPARISON OF TIME WITH PROPOSED METHOD

C. Experimental Study ON APACHE SPARK AND APACHE HADOOP
We have performed whole experimentation on three node
hadoop cluster. Each node has Intel 2.5 GHz i3 processor Dataset Time (sec) on Time (sec) on
having 4 GB RAM. The heap size is set to 1024 M for all spark Hadoop
mapper or reducers. Iris 0.0239 27.403
We have showed results for all datasets described in Table
Landsat 0.0138 62.547
I for basic method (base), the multi-classification approach
(OVA) and the proposed classification system Vehicle 0.0144 32.016
(SMOTE+OVA). The results are summarized in Table II. Waveform 0.0127 24.788
To make more intuitive comparison, we have calculated
f-measure value for testing datasets. The results are
summarized in TABLE III From the above table we can see that Apache Spark gives
We have implemented proposed system on Apache spark better results than Apache Hadoop for same datasets. This is
also. The performance of Apache Spark and Apache Hadoop because Apache Spark processes data in memory while
is measured in terms of time. Here, we have used four Apache Hadoop persists back to the disk after a Map and
datasets from Table I. The results are summarized in TABLE Reduce action, so Apache Spark is faster than Apache
IV. The significant results are in bold face. Hadoop.

2015 IEEE International Advance Computing Conference (IACC) 407

 V. CONCLUSION
In this work, the data preprocessing technique-SMOTE
(Synthetic Minority Oversampling Technique) for multi-class
imbalanced data is presented. Also, we have used RF
(Random Forest) algorithm as a base, which is decision tree
ensemble and known for its good performance. In today‘s
scenario, big data is point of attraction because huge amount
of data that are currently generating. Traditional data mining
techniques are unable to survive with requirements urged by
big data.
We have tested quality of proposed system in terms of
accuracy and F-measure. Experimental analysis carried out
using various datasets of UCI repository. The results obtained
shows that SMOTE+OVA algorithm gives good performance
in the imbalanced data problem. We have tested spark against
hadoop in terms of time spark gives better results as compared
to hadoop.
In future, we will try to address the problems- how to fix
up the oversampling rate as it varies with dataset and how to
generate N synthetic data subsets for each feature vector in the
minority class.
REFERENCES
[1] Apache Hadoop (Accessed in August 2014) http://hadoop.apache.org/
[2] Apache Mahout (Accessed in August 2014) http://mahout.apache.org/
[3] Apache Spark (Accessed in August 2014) http:// spark.apache.org/
[4] Han, J. Kamber, M & Pei, J. (2000). Data Miming: concepts and
techniques (the Morgan Kaufmann Series in data management systems).
Morgan Kaufmann.
[5] Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P.
(2002). SMOTE: synthetic minority over-sampling technique. Journal of
artificial intelligence research, 16(1), 321-357.
[6] Chawla, N. V., Lazarevic, A., Hall, L. O., & Bowyer, K. W., (2003).
SMOTEBoost: Improving prediction of the minority class in boosting.
InKnowldge Discovery in Database: PKDD 2003 (pp. 107-119).
Springer Belin Heidelberg.
[7] Rifkin, R., & Klautau, A. (2004). In defense of one-vs-all classification.
The Journal of Machine Learning Research, 5, 101-141.
[8] Aly, M. (2005). Survey on multiclass classification methods. Neural
Netw, 1-9.
[9] Bunkhumpornpat, C., Sinapiromsaran, K., & Lursinsap, C. (2009). Safe-
408 2015 IEEE International Advance Computing Conference (IACC)
level-smote: Safe-level-synthetic minority oversampling technique for
handling the class imbalanced problem. In Advance in Knowledge
Discovery and Data Mining (pp. 475-482). Springer Berlin Heidelberg.
[10] Batuwita, R., & Palade, V. (2009). microPred: effective classification of
pre-miRNAs for human miRNA gene prediction. Bioinformatics, 25(8),
989-995.
[11] Fernandez, A., Del Jesus, M. J., & Herrera, F. (2010). Multi-class
imbalanced data-sets with linguistic fuzzy rule based classification
systems based on pairwise learning. In Computational Intelligence for
Knowledge-Based Systems Design (pp. 89-98). Springer Berlin
Heidelberg.
[12] Bermejo, P., Gamez, J. A., & Puerta, J. M. (2011). Improving the
performance of Naive Bayes multinomial in e-mail foldering by
introducing distributed-based balance of datasets. Expert Systems with
Applications, 38(3), 2072-2080.
[13] Garci, S., Triguero, I., Carmona, C. J., & Herrera, F. (2012).
Evolutionary-based selection of generalized instances for imbalanced
classification. Knowledge-Based Systems, 25(1), 3-12.
[14] Xiang, H., Yang, Y., & Zhao, S. (2012). Local clustering ensemble
learning method based on improved AdaBoost for rare class analysis.
Journal of Computational Information Systems, 8(4), 1783-1790.
[15] Fernandez, A., Lopez, V., Galar, M., Del Jesus, M. J., & Herrera, F.
(2013). Analysing the classification of imbalanced data-sets with
multiple classes: Binarization technique and ad-hoc approaches.
Knowledge-based systems,42, 97-110.
[16] Hu, F., Li, H. (2013). A novel boundary oversampling algorithm based
on neighboured rough set model: NRSBoundary –SMOTE.
Mahematical Problems in Engineering, 2013.
[17] Han, J., Liu, Y., & Sun, X. (2013, May). A scalable random forest
algorithm based on MapReduce. In Software Engineering and Service
Science (ICSESS), 2013 4th IEEE International Conference on (pp. 849-
852). IEEE.
[18] Lopez, V., Fernandez, A., Garcia, S., Palade, V., & Herrera, F. (2013).
An insight into classification with imbalanced data: Empirical results
and current trends on using data intrinsic characteristics. Information
Sciences, 250, 113-141.
[19] Park, B. J., Oh, S. K., & Pedrycz, W. (2013). The design of polynomial
function-based neural network predicators for detection of software
defects. Information Sciences, 229, 40-57.
[20] Rio, S., Lopez, V., Benitez, J. M., & Herrera, F. (2014). On the use of
MapReduce for imbalanced big data using Random Forest. Information
Sciences, 285, 112-137.