You are on page 1of 18

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/313222794

DDoS Attack Detection and Mitigation Using SDN: Methods, Practices, and
Solutions

Article · February 2017


DOI: 10.1007/s13369-017-2414-5

CITATIONS READS

12 757

3 authors:

Narmeen Bawany Jawwad Shamsi


National University of Computer and Emerging Sciences National University of Computer and Emerging Sciences, Karchi, …
17 PUBLICATIONS   37 CITATIONS    58 PUBLICATIONS   217 CITATIONS   

SEE PROFILE SEE PROFILE

Khaled Salah
Khalifa University
182 PUBLICATIONS   1,138 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Energy Consumption in Cloud Data Centers View project

Arabic reCAPTCHA for digitizing Arabic manuscripts View project

All content following this page was uploaded by Narmeen Bawany on 15 February 2017.

The user has requested enhancement of the downloaded file.


Arab J Sci Eng
DOI 10.1007/s13369-017-2414-5

REVIEW ARTICLE - COMPUTER ENGINEERING AND COMPUTER SCIENCE

DDoS Attack Detection and Mitigation Using SDN: Methods,


Practices, and Solutions
Narmeen Zakaria Bawany1 · Jawwad A. Shamsi1 · Khaled Salah2

Received: 20 August 2016 / Accepted: 9 January 2017


© King Fahd University of Petroleum & Minerals 2017

Abstract Distributed denial-of-service (DDoS) attacks tion and mitigation mechanisms, and we classify them with
have become a weapon of choice for hackers, cyber extor- respect to the detection techniques. Second, leveraging the
tionists, and cyber terrorists. These attacks can swiftly characteristics of SDN for network security, we propose and
incapacitate a victim, causing huge revenue losses. Despite present an SDN-based proactive DDoS Defense Framework
the large number of traditional mitigation solutions that exists (ProDefense). We show how this framework can be utilized to
today, DDoS attacks continue to grow in frequency, vol- secure applications built for smart cities. Moreover, the paper
ume, and severity. This calls for a new network paradigm highlights open research challenges, future research direc-
to address the requirements of today’s challenging security tions, and recommendations related to SDN-based DDoS
threats. Software-defined networking (SDN) is an emerging detection and mitigation.
network paradigm which has gained significant traction by
many researchers to address the requirement of today’s data Keywords Software-defined networking · SDN · DDoS
centers. Inspired by the capabilities of SDN, we present a attacks · OpenFlow · DDoS mitigation
comprehensive survey of existing SDN-based DDoS attack
detection and mitigation solutions. We classify solutions
based on DDoS attack detection techniques and identify
requirements of an effective solution. Based on our findings, 1 Introduction
we propose a novel framework for detection and mitigation
of DDoS attacks in a large-scale network which comprises a Distributed denial-of-service (DDoS) attacks have been a
smart city built on SDN infrastructure. Our proposed frame- real threat for network, digital, and cyber infrastructure
work is capable of meeting application-specific DDoS attack [1].These attacks are capable to cause massive disruption
detection and mitigation requirements. The primary contri- in any information communication technology (ICT) infras-
bution of this paper is twofold. First, we provide an in-depth tructure [2]. There could be numerous reasons for launching
survey and discussion of SDN-based DDoS attack detec- DDoS attacks. These include financial gains [3], political
gains [2], and disruption [4,5]. DDoS attacks can paralyze
B Khaled Salah networks and services by overwhelming servers, network
khaled.salah@kustar.ac.ae links, and network devices (routers, switches, etc.) with
Narmeen Zakaria Bawany illegitimate traffic. They can either cause degradation of
nshawoo@gmail.com service or a complete denial of service resulting in huge
Jawwad A. Shamsi losses. Increasing reliance on Internet and data centers has
jawwad.shamsi@nu.edu.pk aggravated this problem. The growing dependence of crit-
1 ical infrastructure of a country in ICT have given rise to
Systems Research Laboratory, FAST-National University
of Computer and Emerging Sciences, Karachi, Pakistan the need of efficient solutions for protection against DDoS
2 attacks [6,7]. For instance, data centers running critical ser-
Electrical and Computer Engineering Department, Khalifa
University of Science, Technology and Research, vices, such as smart grid, need to be protected in order to
PO Box 573, Sharjah, UAE continue to provide highly reliable services.

123
Arab J Sci Eng

Numerous proprietary and open-source solutions exist balance load in data centers. Motivated by these needs, we
for DDoS attack detection and mitigation. However, these propose an SDN-based proactive DDoS Defense Framework
attacks continue to grow in frequency, sophistication, and (ProDefense) for smart city data center.
severity [8,9]. Rapid detection and mitigation of DDoS The main contributions of this paper are summarized as
attacks has become severely challenging as attackers con- follows:
tinue to use novel techniques to launch DDoS attacks [10].
The rising number of DDoS attacks, coupled with growing • We present an extensive survey of SDN-based DDoS
diversity in their types, causing disastrous impact, has made attack detection techniques. We classify these techniques
DDoS attack detection, mitigation, and prevention the top according to detection mechanisms. Classification allows
most priority. better understanding and improved comprehension of the
For instance, Arbor Networks Inc. [11], one of the lead- existing approaches.
ing DDoS threat protection solutions provider, reported a • We identify pros and cons of each technique and elaborate
334 Gbps attack targeting a network operator in Asia recently. key requirements of an effective DDoS attack prevention
Also, it reported many attacks larger than 100 Gbps globally mechanism.
in 2015 [12]. Many such incidents clearly shows that we • We propose a novel SDN-based proactive DDoS Defense
need new approaches to address the DDoS attack problem. Framework (ProDefense) for smart city data center.
These new approaches must be designed to meet the perfor- ProDefense allows implementation of application-speci-
mance and scalability requirements of modern data centers fic requirements for DDoS attack detection and mitiga-
and provide maximum levels of protection against emerging, tion. ProDefense also has distributed controllers, thereby
complex and elusive, attacks. allowing effective mechanisms for distributing load and
With recent advancements in software-defined network- improving reliability.
ing (SDN) and its rapid and wide-scale acceptance in • We present a case study showing how ProDefense capa-
the network community [13], many researchers have been bilities can be utilized to secure applications built for
actively involved in developing SDN-based network security smart cities.
solutions. SDN-based solutions have attracted more atten-
tion since their adoption in large-scale wide area networks Our work is significant with multiple benefits. For resear-
[13]. The technology enables developers to directly program, chers, it provides a comprehensive analysis of the existing
control, and manage network resources centrally through the work and identifies challenges, whereas for academicians, it
SDN controller. offers a thorough study of the subject. Our work is also use-
SDN offers novel ways to solve long standing network- ful for the developer community in understanding strengths
ing problems, such as routing [14], policy-based network and weaknesses of different solutions. The industrial com-
configurations, and security as discussed in Sect. 3 While munity could also find our work useful in understanding the
security of SDN-based networks has been a point of debate requirements and assessing capabilities of these solutions.
and much literature is available discussing the security of The rest of the paper is organized as follows. In Sect. 2,
SDN infrastructure itself [15–17]. This paper, however, takes we present a brief overview of DDoS attacks and SDN. We
the positive viewpoint on SDN-based security and presents a categorize the existing SDN-based DDoS attack detection
survey of SDN-based DDoS attack detection and mitigation techniques and present a survey of these techniques in Sect. 3.
mechanisms. This is followed by description of DDoS attack mitigation
During the study of existing SDN-based solutions, we strategies in Sect. 4. Further, in Sect. 5, we describe ProDe-
observed that there are many approaches for SDN-based fense which is our proposed framework for detection and
DDoS attack detection. Based on this study, we catego- mitigation of DDoS attack in a smart city data center. In
rized the existing approaches according to their methods of Sect. 6, we discuss SDN challenges and open research areas.
anomaly detection. Further, we noted that there is a need Section 7 concludes the paper and presents the future research
for a proficient DDoS protection framework that can be cus- directions.
tomized with respect to requirements of various applications.
This is specifically true for network infrastructures consist-
ing of heterogeneous applications. For instance, in a smart 2 Background
city, there are multiple applications which require protection
from DDoS attacks. Each of these applications has a differ- The growing prevalence of DDoS attacks show that legacy
ent level of tolerance of network traffic. In addition, each defense mechanisms are only partially effective. List of
application may have different responsiveness requirements recent DDoS attacks on various recognized organizations
as well. Modern data centers also have needs for scalable is provided in Table 1. These attacks are targeting almost
and distributed controllers in order to increase reliability and every organization. Even leading financial institutions and

123
Arab J Sci Eng

Table 1 Today’s popular DDoS attacks


Target Description

Client of US-based security vendor Sucuri [84]. US-based security vendor Sucuri [85] discovered a botnet comprising of CCTV-based
June 2016 botnet used for DDoS attacks. The botnet was generating about 50,000 HTTP requests
per second to the server and occupying its memory with illegitimate traffic
Bank of Greece Web site [86]. May 2016 Series of DDoS attack on Bank of Greece website forced the servers to remain offline
for more than 6 h
HSBC internet banking [87]. January 2016 DDoS attack on HSBC [88], one of the largest banking and financial services
organizations in the world, forced its personal banking Web sites in the UK to shut
down for many hours
Irish government websites [89]. January 2016 Several Irish government websites were temporarily forced offline by an apparent DDoS
attack. The affected Web sites included Central Statistics Office, the Courts Service,
the Health Service Executive, and the Houses of the Oireachtas (parliament)
BBC websites [90,91]. December 2015 The attack on the BBC targeted the main Web site as well as associated services
including the main iPlayer catch-up service and iPlayer Radio app
Thai government websites [92]. October 2015 Several Thai government Web sites have been hit by a suspected distributed
denial-of-service (DDoS) attack, making them impossible to access. Web sites that
were targeted includes site of the ministry of information, communications and
technology (ICT) and the main government Web site
Polish airline [93]. June 2015 Around 1400 passengers were stranded at Warsaw’s Chopin airport when the flight plan
system went down for around 5 hours. The problem was caused DDoS attack—that
generated so many communication requests that it overloaded the server, and it could
no longer carry out its normal functions
Canadian Government Web sites [94]. June 2015 DDoS attack on Canadian government’s computer servers caused federal emails and
several department websites to shutdown
GitHub [90]. March 2015 GitHub, a company that hosts programming repositories, was slammed by DDoS attack.
GitHub’s servers were struck with thousands of illegitimate requests causing
intermittent outages
Client of CloudFlare [25,95]. February 2014 CloudFlare [96], reported an attack reaching to 400 Gbps against one of its customers.
The attackers leveraged a flaw in the network time protocol (NTP) to launch the DDoS
attack
Spamhaus [97]. March 2013 Spam-fighting organization, Spamhaus has been hit by DDoS attack of 300 Gbps

government organizations, having huge IT infrastructure and of compromised hosts (also called zombies) are typically
resources, are unable to encounter such attacks It seems available. Table 2 classifies popular DDoS attacks into three
necessary to explore new paradigms that can successfully categories.
respond to DDoS attacks. SDN has emerged as a potential Prevention against DDoS attacks has been a major focus
candidate for addressing the escalating problem of DDoS of the research community [10,18]. Since DDoS attacks are
attacks. The purpose of this section is to enlighten back- quite frequent, the focus has been to develop a proficient solu-
ground information. First, we briefly describe DDoS attacks. tion, which is capable to effectively detect and mitigate DDoS
This is followed by a description of an effective DDoS attack attacks. In our research, we identify following requirements
defensive mechanism. The section also explains the architec- for an effective DDoS mechanism:
ture and protocols of SDN.
(a) The DDoS protection mechanism must not disturb or
2.1 Distributed Denial-of-Service (DDoS) Attacks disrupt legitimate user’s activities
(b) The DDoS detection mechanism should be able to pre-
The objective of a DDoS attack is to bring down the services vent attacks from within the network as well as from
of a target using multiple sources that are distributed. A typ- outside the network.
ical example of such an attack is the flooding based attack in (c) The mechanism should meet the performance and scal-
which a victim is overwhelmed with the massive amount of ability requirements of modern data centers.
network traffic sent to it. The idea of DDoS attacks revolves (d) The DDoS protection mechanism should be inexpensive
around a fact that a large number of sources distributed across with low deployment cost such as additional hardware.
multiple locations are used to target a victim. Botnets are typi- In addition, the deployment must not require wide-scale
cally useful for launching DDoS attacks as a large collection changes leading to high overhead.

123
Arab J Sci Eng

Table 2 Classification of DDoS attacks


I. Reflection-based flooding attacks

Smurf Attack [98] In Smurf Attack, large number of Internet Control Message Protocol (ICMP) packets
with the intended victim’s forged source IP are broadcast to a computer network using
an IP Broadcast address. This generates huge amount of illegitimate traffic on the
network causing network congestion
Fraggle Attack [98] Fraggle attack is similar to Smurf attack, but it uses illegitimate UDP traffic instead of
ICMP traffic to achieve the same goal

II. Protocol exploitation flooding attacks

SYN Flooding attack [98] In TCP SYN flood attack, an attacker sends the packet with the SYN bit setoff TCP
three-way handshake. The victim responds with the packet back to the source address
with SYN-ACK bit set. The attacker never responds to the reply packet, either on
purpose or because the source address of the packet is forged. Therefore, the victim’s
TCP receive queues would be filled up, denying new TCP connections to legitimate
clients
UDP Fragmentation attack [99] In a UDP Fragmentation attack, attackers send large UDP packets (1500+ bytes) to
consume more bandwidth with fewer packets. The victim’s resources are consumed in
reassembling these forged packets which no ability to be re-assembled

III. Reflection and amplification-based flooding attacks

DNS amplification Flooding attack [100] In domain name system (DNS) amplification flooding attack, the zombies generate
small DNS queries with forged source IP address that generates large volume of
network traffic directed toward the victim
NTP amplification Flooding attack [101] Network Time Protocol (NTP) amplification attack is similar to DNS amplification
attack, but it uses NTP servers instead of DNS servers

(e) The mechanism should be robust, adaptive, and flexible. portable devices, and human weaknesses. Network secu-
(f) Mitigation support should be provided. rity devices that are placed at the edge of network can
(g) Low false-positive and high detection rates are also do least for these internal threats.
desirable characteristic for an effective DDoS protection (c) Emerging trends in networking including cloud and big
mechanism. data, are urging for high bandwidth, high reliability,
ubiquitous accessibility, and dynamic management in
data centers [20]. Ever growing demand and high acces-
The requirements mentioned above can only be consid-
sibility requirements of data makes performance and
ered as guidelines to develop a good DDoS attack protection
scalability a critical challenge.
solution. Severity of each requirements may vary with respect
(d) Maintaining the low false-positive rate may lead to low
to network environment in which it is implemented and
detection rate, such that slow and steady DDoS attacks
the level of protection required. However, there is no sil-
[21] may easily bypass the DDoS attack prevention solu-
ver bullet to eliminate DDoS attacks. Attaining impeccable
tions.
DDoS attack prevention mechanism presents many chal-
(e) Developing an effective open-source DDoS attack pro-
lenges mainly considering the growing complexity of attacks.
tection solution that can compete with the proprietary
Few of the challenges that security researchers have to con-
solutions is challenging because of lack of commercial
sider can be listed as follows:
support [22] along with integration issues with pro-
prietary hardware and software which makes it more
(a) Differentiating between legitimate and illegitimate traf- complicated.
fic is much difficult due to attacks that mimic the
behavior of legitimate network traffic. For instance,
sophisticated botnets are capable of bypassing the DDoS Current DDoS trends are a clear indication of limita-
attack detectors by mimicking the traffic patterns of flash tions of existing technologies. Large-scale volumetric attacks
crowds [19]. are further increasing in size and more sophisticated DDoS
(b) Internal network threats are also growing in an orga- attacks are emerging [23–25]. Attackers have adopted sophis-
nization along with external threats. Internal system ticated mechanisms to bypass the traditional protection
weaknesses include software vulnerabilities, use of shields. More recently, software-defined networks (SDN)

123
Arab J Sci Eng

the communication mechanism that enables the SDN con-


troller to directly interact with the data plane. Controller
pushes packet handling rules in flow tables of OpenFlow
switches. The rule matches the traffic and performs certain
actions, such as dropping, forwarding, modifying on traffic.
Depending upon the rules installed by a controller applica-
tion, an OpenFlow switch can take the role of a router, switch,
firewall, load balancer, etc. [26,31].
SDN is anticipated to be an ideal platform that could con-
tribute largely in developing pragmatic solutions for DDoS
attack detection and mitigation. The notion of separation of
control and data planes, and the concept of flow-based traffic
make detection of attacks much easier. Further, the central
point of control lead to rapid detection [32].
Research community has sought to utilize the distinc-
tive features of SDN in order to enhance security against
traditional cyber attacks, including DDoS attacks. Though,
the SDN paradigm provides capabilities to develop effec-
tive measures to encounter DDoS attack, it also possess the
Fig. 1 SDN architecture risk of vulnerabilities and malfunctions due to attacks on the
controller. These include code injection attack, man in the
middle attack, and denial-of-service attack on the controller.
have emerged as new networking paradigm which have Efficacy of SDN-based networks can be compromised due
received wide-scale attraction. Among the most exciting to these attacks [32]. However, our focus in this paper is
features of SDN are the unprecedented control over net- to enlighten security enhancement features using SDN. In
work infrastructure and decoupling of control and data plane. this context, we present a detail review on existing SDN-
These distinct characteristics of SDN have led to devel- based DDoS attack detection and mitigation solutions. We
opment of many SDN-based DDoS attack detection and also present the classification of DDoS attack detection and
mitigation mechanisms as discussed in the subsequent sec- mitigation solutions based on detection techniques.
tion of the paper. We briefly explain the mechanism of SDN.

2.2 Software-Defined Networking (SDN) 3 SDN-Based DDoS Attack Detection Techniques

Software-Defined Networking simplifies network manage- SDN has many distinctive features which are key in detect-
ment by separating control logic (control plane) from the ing and mitigation DDoS attacks. These features include
underlying hardware that forwards the traffic (data plane). separation of the control plane from the data plane, logi-
With this decoupling of control plane and data plane, network cally centralized controller, programmability of the network
switches become simple forwarding devices whereas control by external applications, software-based traffic analysis, and
logic and functionality are implemented in logically central- capability to dynamically update forwarding rules [33]. We
ized controller. Figure 1 illustrates the basic architecture of describe DDoS attack detection and mitigation techniques
SDN. There are numerous open-source SDN controllers such using SDN in this section. A summary of popular SDN-based
as POX, NOX, Floodlight and OpenDaylight [26,27]. It is DDoS attack detection techniques is given in Table 3.
pertinent to mention that a logically centralized controller
in SDN does not necessarily imply a physically centralized 3.1 Entropy
system [26]. In fact, many SDN frameworks, support dis-
tributed frameworks such as ONOS [28] and OpendayLight Entropy is used to measure the randomness of an attribute
[29]. Efficacy of SDN has been established as it has been within a given period of time. Entropy-based techniques have
adopted by organizations, such as Google to manage Wide been established as an effective approach to compute the ran-
Area Network [13]. Distinguishing features of SDN makes domness of a dataset [34]. High entropy values signify a more
it a strong candidate for next-generation Internet architec- dispersed probability distribution, while low entropy val-
ture [18]. ues denote concentration of a distribution. Therefore, these
OpenFlow [30] protocol is the first and most widely techniques are used for anomaly detection extensively in tra-
deployed protocol on SDN. The OpenFlow protocol defines ditional intrusion detection systems [35–37]. This approach

123
Arab J Sci Eng

Table 3 Summary of popular SDN-based DDoS attack detection techniques


Techniques Description

Entropy [34,41,42] Entropy-based methods depend on network feature distributions to detect anomalous
network activities. Probability distributions of various network features such as source
IP address, destination IP address, and port numbers are used to calculate the entropy.
Predefined thresholds on changes in the entropy values are used to identify the
presence of anomolies
Machine learning [55,56,58,59] Machine learning-based methods employ techniques such as Bayesian networks, SOM,
and fuzzy logic to identify the presence of anomolies. These algorithms takes into
account various network features and traffic characteristics to detect the presence of
anoamlies
Traffic pattern analysis [61,63] These techniques work on the assumptions that the infected hosts exhibit similar
behavioral patterns which are different from benign hosts. Typically, in case of a
botnet attack, infected machines (bots) are usually controlled by single bot master.
Similar traffic patterns are observed as a result of command that is sent to many
members of same botnet causing the similar behavior (e.g., sending illegitimate
packets, starting to scan)
Connection rate [42,56,57,61,63,66] These techniques are classified into two types: 1) connection success ratio and 2)
connection rate, where connection rate refers to the number of connections instantiated
within a certain window of time. See Table 4
SNORT and OpenFlow integrated [71,72] These technique use combination of intrusion detection system (such as SNORT) and
OpenFlow to detect attacks and reconfigure the network dynamically. An intrusion
detection system monitor the traffic to identify malicious activities. OpenFlow
switches are then dynamically reconfigured based on the detected attacks in real time

is more successful in getting the fine grained patterns that A distributed algorithm for entropy-based anomaly detec-
typical volume-based traffic analysis cannot capture [38– tion scheme was proposed by Wang et al. [41]. In that, DDoS
40]. Entropy can be computed on several features such as flooding attack detection module runs in every OpenFlow
network flows, IP addresses, and number of packets. These edge switch thereby reducing the flow collection overload
techniques have the advantage of having a low calculation on the controller. Probability distribution of destination IP
overhead [41]. Due to success of entropy-based algorithms address at each switch is used to calculate the entropy. Once
in traditional network in detecting DDoS attacks, they a DDoS flooding attack is detected, an alert information is
have effectively been employed in SDN-based networks as sent to the controller.
well. While entropy-based techniques can be used to detect
Mehdi et al. [42] uses maximum entropy estimation [36] DDoS attacks, they suffer with a few limitations. Typically,
to estimate the benign traffic distribution to detect network entropy-based detection schemes detect unexpected changes
security problems in home and office networks using SDN. in the time series of the entropy of certain traffic features. The
Traffic is divided into packet classes and maximum entropy probability distribution of a feature is represented by a single
estimation is then used to develop a baseline benign distri- value when entropy is calculated. This is very effective for
bution for each class. Packet classes are based on protocols analysis; however, relevant information about the distribu-
and destination port numbers. Experiments were conducted tion of the analyzed feature is lost. This leads to masking of
using OpenFlow switches and a NOX controller. However, anomaly effects in some cases [45]. Similarly, the different
the authors only used the low rate network traffic to do the distributions with the same amount of uncertainty cannot be
experiments as they are focused on a home environment. distinguished by entropy values. Hence, the anomalies which
Giotis et al. [34] implemented a widely used entropy- do not disturb randomness remain undetected [46].
based approach [43] to effectively detect DDoS, worm
propagation, and portscan attacks. The flow-related traffic 3.2 Machine Learning
features used to detect anomolies are source and destination
IP addresses and ports. Predefined thresholds on changes Machine learning-based techniques are widely applied in
in the entropy values are used to identify the presence of tradational intrusion detection systems (IDS) [47,48]. Arti-
anomolies. Moreover, the authors have used sflow [44] proto- ficial neural networks [49], Bayesian networks [50], self-
col that reduces communication between OpenFlow switches organizing map (SOM) [51], and fuzzy logic [52] principles
and the controller. However, flow rate sampling may affect and concepts are extensively used for detecting anomalies
the accuracy of anomaly detection schemes [41]. in IDS. They have been considerably successful and are

123
Arab J Sci Eng

implemented widely both in wired and wireless networks lected for certain intervals. These flow statistics are collected
[53]. Likewise, these techniques are effectively applied in from OpenFlow switch by the controller. Anomalies are
SDN-based DDoS attack detection. In general, a machine spotted by comparing the expected deviation and the real
learning-based technique differentiates the network flows deviation in the dataset. The authors have presented a three
based on certain features related to traffic characteristics and phase solution for DDoS attack detection and mitigation. In
categorizes them as malicious and benign [54]. the first phase, anomaly in network flow is detected. This
Self-organizing map, one of the most popular neural net- is then followed by identification of sources through packet
work models, is used by Braga et al. [55] for detection analysis of samples in the second phase. The third phase
of DDoS attacks. This work presents SDN-centered DDoS drops all flows coming from malicious sources. The solu-
attack detection based on six traffic flow features. These fea- tion was deployed on a controller developed using Ryu SDN
tures include Average of Packets per flow (APf), Average of framework.
Bytes per flow (ABf), Average of Duration per flow (ADf), Machine learning-based techniques are preferred to detect
Percentage of Pair-flows (PPf), Growth of Single-flows (GSf) suspicious activity based on the abnormal behavior of the
and Growth of Different Ports (GDP). The features are col- network. However, the performance of these techniques is
lected by a flow collector module in a NOX based network typically dependent upon the dataset that has been used for
and are passed to the classifier module for detection of ille- training.
gitimate flows. Self-organizing map is used for flow analysis.
This work shows successful DDoS attack detection with high 3.3 Traffic Pattern Analysis
rate of detection and very low rate of false alarms. Exploit-
ing the capability of SDN for software-based traffic analysis, This technique works on the assumptions that the infected
the method incurs low overhead when compared to tradi- hosts exhibit similar behavioral patterns which are differ-
tional counterparts. Major reason for this low overhead is ent from benign hosts. Typically, in case of botnet attacks,
that traditional approaches collect and process every packet. infected machines (known as bots) are usually controlled by
However, in flow-based approach, samples are collected after single bot master. Similar traffic patterns are observed as a
every three seconds without hammering the rate of attack result of command that is sent to many members of same
detection. Though, this work proposes an efficient DDoS botnet causing the similar behavior (e.g., sending illegiti-
attack detection method it does not discuss any mitigation mate packets, starting to scan) [60]. Foreseeing the SDN
strategy. Further, experiments are conducted on limited scale platform’s success in providing security solutions, Shin et
with small topology using single controller al. [61] presented a comprehensive framework—FRESCO,
Dotcenko et al. [56] proposed a security mechanism for development of OpenFlow-based security applications.
that uses the combination of rate limiting and TRW algo- Several applications are implemented to demonstrate the
rithms [57] with fuzzy logic inference. Mamdani algorithm pragmatism of the FRESCO framework. For instance, Bot-
was used to carry out fuzzy inference. This is considered as Miner is an application which is built upon the FRESCO
one of the most pragmatic approach in solving fuzzy mod- framework [62].The application assumes that hosts infected
eling problems as it is based on fuzzy logic, and it avoids with the same botnet exhibit similar patterns at the network
excessive computation. Results show that decision making level, and these patterns are different from benign hosts.
based on fuzzy rules is better than using the security algo- Hosts with similar packets per second and bytes per second
rithms, separately. However, experiments were carried with are marked as potential threats. Jin and Wang [63] proposed
low traffic and no mitigation strategy has been presented this a system to detect mobile malware by identifying suspicious
work. network activities through real-time traffic analysis. One of
NICE [58] is an intrusion detection framework to pre- the algorithm presented in this work detects the infected hosts
vent vulnerable virtual machines from being compromised by identifying aggregates of similar communications. Com-
in the cloud. NICE uses a novel attack graph approach mon destination, common connection time, and common
for attack detection and prevention by correlating attack platform features were used to extract aggregates of malware
behavior. Attack graph in NICE is constructed based on mul- communications.
tiple attributes including cloud system information, virtual
network topology, configuration information, and vulnera- 3.4 Connection Rate
bility information. When vulnerabilities are discovered and
attack is identified, several countermeasures including, traf- Connection rate-based anomaly detection techniques can be
fic redirection, traffic isolation, port blocking, and network further classified into two types. Connection success ratio and
reconfiguration can be taken. number of connections established as summarized in Table 4.
Dillon et al. [59] proposed anomaly detection algorithm Such that the former denotes ratio of successful connections
that computes standard deviation of packet/byte rate col- over the overall connections, whereas the latter represents the

123
Arab J Sci Eng

Table 4 DDoS attack detection using connection rate


Connection rate-based techniques

Connection success ratio [42,57,61,63] These techniques are based on the probability of a connection attempt being successful
should be much higher for a benign host than a malicious host. Whenever the
likelihood ratio for a host exceeds a certain threshold, it is declared as infected
Number of connections established [42,56,63,66] These techniques are based on the notion that an infected machine attempts to connect to
many different machines in a short period of time. On the other hand, an uninfected
machine makes connections at a lower rate and is more likely to repeat connection
attempts to recently accessed machines. It uses a threshold to limit the number of new
connection attempts within a certain time period

number of connections instantiated within a certain window ing acceptance in traditional networking environments, we
of time. could not find any SDN-based solution that utilizes Suri-
Connection success ratio technique takes into account cata. However, the wide acceptance of SNORT in network
that the probability of successful connection is much higher community and increasing recognition of SDN have led
for a benign host as compared to malicious host [57]. For to development of systems that implements the combina-
each host, the algorithm maintains a list of new connec- tion of OpenFlow and SNORT. Snortflow [43] is one such
tion requests, like TCP SYNs, which have not received a system that integrates both open flow and SNORT. The sys-
response, that is, SYN-ACK. If any of these connection times tem enables cloud system to detect intrusions and deploy
out or receives a TCP RST, the algorithms increases the like- countermeasures by reconfiguring the cloud networking sys-
lihood ratio of the host being infected. The threshold random tem on-the-fly. Similarly, NICE [58] presents a multi-phase
walk with credit-based rate limiting (TRW-CB) algorithm is distributed vulnerability detection, measurement, and coun-
typically used for detecting anomalies on the basis of con- termeasure selection mechanism. It utilizes the capabilities
nection success ratio. TRW-CB algorithm [57] has also been of both SNORT and OpenFlow to detect an intrusion in
incorporated by SDN-based DDoS attack detection schemes cloud-based system. In particular, the solution is aimed at
[42,56,63]. Experiments have been conducted using various preventing virtual machines from being compromised in
controllers which include using NOX controller [42], Bea- cloud-based system.
con Controller [56], and Floodlight controller [63]. All results Once the DDoS attack is detected, we need mitigation
show that attack detection has been successful. strategies to ensure continuity of services. Effective DDoS
Likewise, connection rate-based technique assumes that attack protection solution is incomplete without the mitiga-
number of connection attempts from a compromised machine tion capabilities. Leveraging the capabilities of SDN various
to a server or other machines is much higher than benign DDoS attack mitigation functionalities are incorporated in
hosts. In general, benign hosts make connections at a lower DDoS protection solutions as discussed in next section.
rate and are expected to repeat connection attempts to
recently accessed machines. It utilizes a threshold to limit
the number of new connection attempts within a certain 4 SDN-Based DDoS Attack Mitigation Techniques
time period [64,65]. These algorithms have also been imple-
mented on many SDN controllers such as NOX [42], Beacon SDN has been focused to improve agility and flexibility
controller [56], Floodlight controller [63], and POX con- of a network. It empowers networks to respond quickly to
troller [66]. changing network requirements via a centralized controller.
The SDN controller provides a global view of the network.
3.5 Integration of SNORT and OpenFlow Further, the notion of the centralized controller, leads to
consistent configuration throughout the network. Since, all
SNORT [67] has been a popular open-source network intru- network policies are defined at a centralized controller, it
sion prevention system (NIPS) and network intrusion detec- not only simplifies anomaly detection, but also facilitates
tion system (NIDS) since long. Recently, Open Information in prompt invocation of mitigation mechanism [70]. For
Security Foundation (OISF) has developed Suricata [68] as example, when a DDoS attack is detected, a threat mitiga-
an alternative to SNORT. Suricata, also an open-source net- tion application may effectively reprogram switches to block
work security solution, has been termed as a next-generation malicious traffic.
NIDS [69]. It supports multi-threaded operations that makes Many SDN-based DDoS defense mechanisms incorpo-
it more efficient as compared to SNORT that typically is a rates mitigation strategies. Dropping packets, blocking ports,
single threaded system. Though Suricata appears to be gain- and redirecting traffic are commonly used mitigation strate-

123
Arab J Sci Eng

Table 5 Summary of prevalent DDoS attack mitigation techniques


Attack Mitigation techniques

Drop packets [34,59,71] The network traffic conforming to defined rules is transmitted and remaining is dropped
Block port [58,72] The network traffic from attacking port is completely blocked
Redirection [58,66,72] The legitimate traffic is redirected to new IP address
Control bandwidth [74] The controller limits the flow transmission rate by allocating average bandwidth to each
interface
Network reconfiguration and topology change [58] The network controller changes the flow table on each switch to change the network
topology
Deep Packet Inspection [58,72] Deep packet inspection is a process that may completely examine both header and data
part of packet. Deep packet inspection enables security functions and makes it possible
to detect several types of attacks including buffer overflow attacks, denial-of-service
attacks, and worms and virus attacks
MAC address change and/or IP address When the attack is detected MAC address or IP address of the victim is changed.
change [58,72] Legitimate traffic is routed to new address and malicious traffic it blocked
Quarantine or Traffic isolation [58,72] This mitigation technique prevents the network resources from being overwhelmed by
the volume-based attack by isolating the malicious traffic

gies that are extensively deployed in SDN [34,58,59,66,71, attack is detected. However, alternate paths can be utilized
72]. Nevertheless, other mitigation strategies such as deep for legitimate users.
packet inspection, changing MAC and IP addresses, and All the above-mentioned DDoS defense mechanisms are
isolating traffic are also implemented in SDN-based DDoS based on computing a threshold value that serves as a baseline
defense mechanisms [58,72]. Brief description of prevalent of for attack detection. This threshold value remains constant
DDoS mitigation techniques is summarized in Table 5. and consistent for all the applications. That is, it cannot be
Dropping packets and blocking ports are simple and fast customized with respect to different applications utilizing the
mitigation techniques that completely block the potential network. However, in modern data centers, there are diverse
attack sources. These mitigation techniques may result in applications with different requirements and expectations of
dropping the legitimate traffic. For instance, if a legiti- tolerance of network interruption and service.
mate node or port is compromised, it is completely blocked For instance, there are certain critical applications with
resulting in dropping of any legitimate traffic it may per- high requirements of uptime and availability. For such appli-
haps generate. Similarly, in case of false attack detection, cations, rapid detection of an attack is critical. In contrast,
legitimate users are barred from using services. The traffic there are certain applications with comparatively moderate
redirection technique forwards the traffic to new IP address requirements of uptime. These applications can tolerate some
after the attack is detected on an existing server. Initially, all delay in network availability, and they can also afford high
connections to the existing server are torn down in order to false positive which may lead to service denial to legitimate
prevent bots from directly accessing the redirected address. users. Consequently, in case of a large-scale data center run-
The redirected address either imposes high computation bar- ning diverse applications, a customizable solution is needed
rier for bots, such as using captcha [73], or imply deep packet with varying degree of sensitivity to possible attacks [7,25].
inspection [58,72] to analyze and forward only legitimate We believe that the above-mentioned requirement is a
traffic. Nonetheless, this technique increases the processing highlighting factor for DDoS detection. While current SDN-
time of the network traffic along with the overall complex- based solutions provide improved flexibility over conven-
ity. Attack mitigation by controlling bandwidth [74] at each tional DDoS attack detection solutions, they do not provide
router interface prevents user’s starvation. Flows exceed- application-specific detection and mitigation.
ing the assigned average bandwidth are penalized, such as In addition, scalability has now become an intrinsic
experiencing delays. This mitigation strategy benefits low requirement for DDoS attack prevention solutions. Continu-
bandwidth flows, which cause low impact on the network ous increase in massive DDoS attacks rationalize the need of
and have short duration. However, legitimate flows may suf- scalability. In an SDN solution, scalability implies the ability
fer if they do not adhere to network usage policies. Network of the network to handle large-scale traffic. For this purpose,
topology change strategy [58] is very effective in SDN-based a distributed controller architecture with capabilities of man-
DDoS attack mitigation. This counter measure ensures that aging large amount of traffic is required. However, existing
the path of an attack is disconnected immediately after an SDN solutions are focused only on a single controller, which

123
Arab J Sci Eng

limits the capabilities of the network. A distributed controller We use smart city as a case study to highlight the versatility
framework also improves the availability of the controller in and key features of the ProDefense framework. In such a city,
case of controller failure [75]. operations and services such as traffic control, power distri-
Motivated by the above-mentioned needs, we propose bution, transport, and other utilities are monitored, operated,
ProDefense framework that addresses the aforementioned and controlled through ICT based infrastructure, smartly. A
requirements. Also, we present a case study of smart city smart city system has many distinct characteristics. It is a
data center to show the diversity in security requirements of highly complex system due to its enormous structure and
various applications. heterogeneity. Moreover, smart city comprises of extremely
critical control systems. These systems are so vital that their
destruction or service disruption can affect safety, secu-
rity and economy of the city. Consequently, smart city can
5 SDN-Based Proactive DDoS Defense Framework become a prime target for terrorism and cyber attacks because
(ProDefense) of the critical nature of services that it provides. Adversaries
can target smart city networks and application servers to par-
In this section, we present a customizable DDoS defense alyze the city almost instantly. Therefore, cyber security is
framework, named as ProDefense, which generates DDoS one of the major challenges in a smart city system. Secur-
attack alerts considering the application’s security require- ing smart city data centers from cyber attacks and ensuring
ment. ProDefense has been motivated by our analysis that continuity of services is utmost desirable.
different applications have different security requirements. We argue that smart city network requires a highly cus-
Therefore, DDoS attack detection solution must include tomizable security solution to accommodate the need of
a customizable reaction mechanism for generating DDoS diverse smart city applications. ProDefense framework is
attack alerts to match the application-specific needs as dis- well suited for smart city network as it caters the need of
cussed in previous section. application-specific security requirements. Numerous smart
ProDefense leverages the programming and dynamic city applications can be developed to facilitate masses.
nature of SDN and implements an adaptive DDoS protection These includes e-government, smart transport system, smart
mechanism. Figure 2 depicts the architecture of the proposed grid, smart healthcare, and smart rescue services. These
framework. The distributed controller layer increases the reli- applications can be divided into three categories, namely,
ability and scalability of the solution to live up to the needs catastrophic, critical, and moderate, with respect to their
of a huge data center. cyber security requirements as shown in Table 6.

Fig. 2 Architectural design of


ProDefense

123
Arab J Sci Eng

Table 6 Categories of Smart city applications and related security aspects


Applications Impact of DDoS Application security Security solution requirement ProDefense Filters
attack requirements

Smart Grid, Catastrophic Such applications needs to Extremely Agile HR filter


Traffic Control trigger the mitigation system As the impact of attack is
System immediately. The security highly critical such applications
alert is generated foreseeing requires very low false-negative
the malicious behavior before rate of attack detection
reaching the threshold. The The security solution must take
security solution monitors the the proactive approach
network traffic trends and
predict the attack
Healthcare, Critical Such application trigger the Agile IR filter
Location-based mitigation system within a As the impact of attack is
services certain time period after the critical such applications needs
attack is detected an adequate solution that
neither reacts before time nor
delays the alert
Weather update, Moderate Such applications delay the Moderately agile LR filter
News, Parking invoking of the mitigation As the continuity of services is
system. Only when the attack more important such
is confirmed the mitigation applications require very low
system is triggered. These false-positive rate of attack
applications cannot afford to detection. The security solution
block legitimate traffic takes the reactive approach
unnecessarily

The applications providing crucial services, such as Smart and least reactive (LR) filter for the applications in catas-
Grid, have stringent requirement for cyber security therefore trophic, critical, and marginal categories respectively.
need an attack detection mechanism having very low false- HR filter takes the value of α = 0.1 in Eq. (1) which
negative rate. Understandably, these services cannot afford causes the effect of current traffic rate to be dominant. That
a downtime therefore an extremely agile security solution is is, HR filter immediately generates security alerts. This filter
needed that takes proactive approach. The detection and mit- can generate more false positive as little spikes in network
igation solutions must go an extra mile to fulfill the require- traffic are also taken as threats. This type of filter provides
ments of applications in catastrophic category. In contrast, early warning mechanism which is suitable for highly critical
applications providing information services to citizens such networks such as smart grid.
as weather, news, and sports have less stringent cyber secu- An IR filter takes the value of α = 0.5 in Eq. (1). This
rity requirements. These applications needs a solution with makes it to takes into account both the current rate of traffic
very low false-positive rate hence a moderately agile security and the previous rates. For most networks, this filter will
solution seems appropriate in this case as shown in Table 6. be appropriate as it equally takes into account current and
Motivated by the use of exponentially weighted moving previous traffic rates
average (EWMA) filters [67,68], ProDefense utilizes such The LR filter is the most stable filter that reacts much
filters to customize the attack detection. Equation (1) shows slowly to attacks as it takes the value of α = 0.9 in Eq. (1).
the modified form of equation of EWMA filter used in ProDe- This filter is more useful where cost of false positive is higher.
fense. This filter is a deliberate attempt to lower down the false-
positive rate. In case of smart city application this filter is
PTt = αPTt−1 + (1 − α) CTt + c (1) suitable for applications that require less agility.
The design of our SDN-oriented ProDefense framework
where: PTt = predicted traffic, CTt = current traffic, α is modular. It has three major components, namely traffic
= gain, c constant dependent on traffic characteristics. flow collector, policy engine, attack detector, and mitigation
Typically, these filters are either able to react quickly engine as depicted in Fig. 3. The mitigation engine, adaptive
in case of attack or respond slowly, masking the transients filters, and threshold detector functions are decoupled and
depending upon the value of α known as the gain. Lower can be adapted to future security requirements. Traffic flow
values for α will generate DDoS alert almost instantly. We collector collects the flow entries from OpenFlow switches.
present three different variants of these filters. These are The policy engine is used to define attack detection policy
highly reactive (HR) filter, intermediate reactive (IR) filter, and mitigation policy. For attack detection, policy engine is

123
Arab J Sci Eng

(e) It is a lightweight application which does not require


high computation.
(f) The adaptive filter module employed by ProDefense
helps in rapid detection of attacks. Further, it allows
meeting application-specific requirements.
(g) The ProDefense framework supports various mitigation
strategies such as blocking port, diverting flows, and
controlling bandwidth. Using various configuration of
filters, ProDefense can be customized to achieve essen-
tial accuracy of attack detection.

The ProDefense framework is a novel approach toward


securing smart city data centers against DDoS attacks.
Application-specific customization serves works as an early
warning system to protect critical city services and infras-
Fig. 3 ProDefense framework tructure from cyber criminals. Incorporation of mitigation
module ensures continuity of city services to maximum
extent.
used to configure the type of filter to be used by the network
for detecting attacks. Three types of filter can be configured,
that is HR filter, IR filter, and LR filter. For attack mitigation, 6 Open Research Challenges, Future Research
policy engine defines the defense strategy. ProDefense sup- Directions, and Recommendations
ports many types of defense strategies including dropping
packets, blocking ports, and traffic redirection. The attack SDN promises programmability and centralized control that
detector module takes the input from Traffic Flow collec- enables an adaptive and efficient network. SDN has been
tor and generates security alerts with respect to the policy prominently termed as a technology behind future networks.
defined in Policy Engine. These security alerts trigger the However, there are several challenges which are needed to
mitigation module for taking appropriate action. be catered. We highlight some of these crucial challenges
The ProDefense framework has been designed to meet below where SDN community needs to exploit its research
the requirements of an effective DDoS defense mecha- potential.
nism (Sect. 2.1). The framework is expected to immediately
respond to attacks and ensure continuity of operations 6.1 Adoption of SDN
through integrated detection and mitigation mechanisms.
This is specifically useful in the context of smart city, where Adoption of SDN is one of the foremost challenge that need
applications requirements may vary. In addition, the frame- to be taken into account. A major requirement is the deploy-
work allows customization of policies through filters to fulfill ment of SDN-enabled hardware, which adheres additional
the need of various applications. The key features of ProDe- cost. Further, administration of SDN may require enhanced
fense can be summarized below: training to design, maintain, and operate networks. Debug-
ging and maintaining SDN-enabled networks may also be
(a) ProDefense can filter attacks traffic by sending suspi- non-trivial due to lack of experts [20]. These problems have
cious traffic to controller for inspection and allowing the been addressed using hybrid SDN models [79] that help in
legitimate flows to pass through the network without dis- partial transition to SDN. These models integrate SDN with
ruption. traditional network. However, transition from traditional net-
(b) It is capable to monitor all traffic flows. This would works to SDNs remain an open research area that need more
allow the framework to detect anomalous behavior of efforts from the research community.
both internal and external nodes.
(c) ProDefense utilizes distributed controllers. This would 6.2 Availability of Production-Level SDN Controllers
allow the framework to meet scalability, performance,
and reliability requirements. Availability and reliability of a production-level SDN con-
(d) ProDefense does not require any additional hardware and troller is another major challenge [79]. Despite rapid research
can be easily deployed as an application running on top growth and technological advancements to build effective
of existing SDN controller. SDN controllers, availability of SDN controllers with proven

123
Arab J Sci Eng

technical support remains a barrier in deployment of SDN unknown packets to OpenFlow switches [80,81]. Within
on a large scale. The community needs SDN controllers that SDN, whenever a packet reaches an OpenFlow switch, it
can match the performance of their traditional counterparts is forwarded according to the rule specified in the flow table
in terms of maintenance, security, availability and reliability. of the switch. In case no matching flow entry is found, the
Major efforts are required in the form of open-source soft- packet is forwarded to the controller for further action. The
ware as well as commercial license products. Development controller then decides how to handle the packet and installs a
of distributed SDN controllers meeting the requirements of new rule in the OpenFlow switch, if required. With large vol-
scalability, availability, and reliability of large-scale data ume of such traffic coming in the SDN controllers direction
centers is also an issue, which require immediate attention can not only bring down the controller, but also absorb high
from the research community. Further, the distributed SDN bandwidth of the communication channel between switches
controller architecture—where number of SDN controllers and controller. SDN controller can become a performance
are created to enable automatic failover and load balanc- bottle neck and may not work efficiently. Improving the effi-
ing, requires consistent updates throughout the network. The ciency and performance of SDN controllers so that they can
issues related to synchronization of these controllers while scale up with increasing number of OpenFlow switches is a
maintaining the low overhead is a matter of open debate. complex task that needs to be addressed effectively.

6.3 Security of SDN Controller 6.6 Identifying Malicious Traffic

Inherently, SDN has a centralized control plane, which Most of the techniques used to detect DDoS attacks are
empowers it for improved control and management. How- based on some threshold value. The alerts are generated
ever, this central management feature can also become a when network traffic crosses the maximum allowed thresh-
security threat for SDN such that the SDN controller is also old limit. Therefore, if illegitimate traffic remains below the
a vulnerable target for attackers. This is specifically true in threshold, it remains undetected. Sophisticated attacks that
the absence of a robust and secure controller platform [17] resembles the traffic patterns of legitimate users are also
In this context, conducive efforts are needed in exploring difficult to detect. These include flash crowd-based attacks
innovative methods for ensuring security of an SDN con- [82] that can mimic legitimate behavior in order to bypass
troller. Similarly, communication channels between switches the defense mechanism easily. Typically, threshold-based
and controller must also be secured. Any attack on the com- mechanisms activates mitigation mechanisms when network
munication link between the controller and the switch can traffic exceeds a specific value. This in turn causes the degra-
cause a considerable damage to the network [17]. Develop- dation or blocking of services to legitimate users. Hence,
ment and implementation of effective security specification more reliable and effective mechanisms are required to dif-
for the controller–switch interface is also the subject of con- ferentiate between legitimate and illegitimate traffic so that
cern for the SDN community. malicious only users can be identified and blocked.

6.4 Real-Time Network Traffic Monitoring Overhead 6.7 Vulnerabilities Due by SDN Applications

Monitoring of network traffic is vital for network manage- SDN separates control plane from data plane which facilitates
ment. It also provides basis of network security solutions. high level network abstraction and programmability. This key
Specifically, DDoS attack-based security solutions requires feature opens the SDN network to applications that can be
real-time monitoring of network traffic to generate security used to implement innovative network functions. This can
alerts leading to timely mitigation of attacks. Existing mon- lead to installation of applications with conflicting rules [83].
itoring solutions such as sflow [34] imposes considerable Also malicious applications can be installed that can bring
overhead that reduces efficiency of security applications in down the controller. Various mechanisms that can effectively
large-scale networks. Though frequent collection of statistics inspect SDN applications before executing them are needed.
from network devices to controller increases the overhead Likewise, securing controller from vulnerabilities caused by
it results in high accuracy. Trade-off between reducing this application is an area of further study
overhead, while maintaining high accuracy, is an important The above-mentioned analysis provides a holistic view
direction. about SDN and associated challenges in the adaptation.
Understandably, professionals and practitioners are required
6.5 Overwhelming Controller with Network Traffic to meet the growing demand and challenges of evolving
SDN domain. It also highlights a few significant research
SDN controllers are also susceptible to malicious attacks. directions. Most importantly, the research community should
SDN controllers can be overwhelmed merely by sending play an active role in ensuring the security of SDN platform.

123
Arab J Sci Eng

These include security of the controller and associated links.. controller platform which allows load balancing and reduces
Thorough efforts are also required to incorporate and adopt possibilities for controller failure.
SDN-based security solutions. In this context, significance ProDefense is a major step in resolving the above-
of cost-effectiveness is also important in global adaptation mentioned issues. We anticipate that ProDefense can poten-
of SDN. The developer community should play a leading tially be utilized in a wide range of systems including cyber
role to meet these expectations. The role of computer sci- physical systems, smart grid, and e-governance. All such
entists is also important in determining balance between a applications exhibit varying degree of tolerance for network
highly reactive threshold-based detection scheme vs a con- attacks. ProDefense would require to implement consistency
servative framework which may allow sudden increase in among all the controllers in order to ensure rapid incor-
network traffic. poration of network rules. ProDefense framework can be
implemented in any programming or scripting language that
is compatible with the SDN controller being used. However,
7 Concluding Remarks we are implementing the key modules of ProDefense such as
threshold detector and adaptive filter module using node.js.
The increased reliance on cyber physical systems and We selected node.js because it is an asynchronous IO model
advancements in networking and cloud technologies have which supports parallelism, allowing the SDN application to
highlighted the need for protection of network and comput- interact with distributed controllers and monitoring systems
ing infrastructure against DoS attacks. However detection without blocking [76]. This will result in a fast and consistent
and mitigation of DDoS attacks has remained an unaccom- response time. For implementation of flow collector module,
plished task. Realizing this open-ended problem, we have we are using sflow [44] to collect and monitor network traffic
made several noticeable contributions. statistics. Using sflow provides a number of advantages. It
In this paper, we have discussed major requirements and is a lightweight sampling technology with no performance
challenges in meeting this uphill task of prevention against overhead in switches and routers. Moreover, the sFlow archi-
DDoS attacks. An effective DDoS attack detection sys- tecture not only scales better but it is also supported by most
tem requires the ability to rapidly respond to increasing switch vendors [77]. As a future step of our research, we
malware traffic. Furthermore, customizability and ease of are planning to perform comprehensive experiments using
management in application of rules for detection and pre- ProDefense to study the effectiveness of the framework in
vention against DDoS attacks is also required. Considering protecting Smart city applications. Lastly, SDN is not a sil-
the emerging potential of SDN in meeting network-wide ver bullet solution to all network security problems. In this
requirements of flexibility, management, and adaptability, paper, we also highlighted open research issues, challenges,
we assessed it’s capabilities against mitigation and preven- and recommendations related to SDN-based DDoS attack
tion of DDoS attacks. We classified existing SDN-based detection and mitigation that require further research [78].
solutions according to various detection and mitigation tech-
niques and highlighted pros and cons against each category.
Our analysis revealed that existing SDN-based solutions for References
DDoS detection and mitigation failed to meet application-
specific requirements for DDoS attack detection. In that, 1. Geng, X.J.; Whinston, A.B.: Defeating distributed denial of ser-
traffic threshold for DDoS attack detection can vary for each vice attacks. IT Prof. 2(4), 36–42 (2000)
2. Ottis, R.: Analysis of the 2007 cyber attacks against Estonia from
application; however, existing solutions do not implement the information warfare perspective. In: Proceedings of the 7th
any mechanism to cater this important need and implement European Conference on Information Warfare, p. 163 (2008)
same level of threshold for each application. Further, existing 3. Bangladesh Bank heist. (2016). https://en.wikipedia.org/wiki/
solutions utilizing SDN for DDoS attack detection and miti- 2016_Bangladesh_Bank_heist
4. European renewable power grid rocked by cyber-attack. EurAc-
gation incorporates a single controller. This not only creates tiv (2012). https://www.euractiv.com/section/energy/news/
a bottleneck for the network traffic but also leads to single european-renewable-power-grid-rocked-by-cyber-attack/
point of failure for the SDN network. Although, SDN can 5. Musil, S.: Record-breaking DDoS attack in Europe hits
incorporate distributed platform but existing solutions have 400 Gbps. CNET (2014). http://www.cnet.com/news/record-
breaking-ddos-attack-in-europe-hits-400gbps/
not realized it to its full potential. 6. Paroutis, S.; Bennett, M.; Heracleous, L.: A strategic view on
We have addressed the two problems by proposing an effi- smart city technology: the case of IBM Smarter Cities during a
cient system for DDoS attack detection and mitigation. Our recession. Technol. Forecast. Soc. Chang. 89, 262–272 (2014)
proposed framework, ProDefense, incorporates application- 7. Bawany, N.Z.; Shamsi, J.A.: Smart city architecture: Vision and
challenges. Int. J. Adv. Comput. Sci. Appl. 6(11) (2015)
specific criteria for network traffic threshold. This permits 8. Yadav, V.K.; Trivedi, M.C.; Mehtre, B.M.: DDA: an approach to
implementation of customizable criteria for detection of handle DDoS (Ping flood) attack. Adv. Intell. Syst. Comput. 408,
DDoS attacks. In addition, ProDefense utilizes distributed 11–23 (2016)

123
Arab J Sci Eng

9. Saied, A.; Overill, R.E.; Radzik, T.: Detection of known and 30. McKeown, N.; Anderson, T.; Balakrishnan, H.; Parulkar, G.;
unknown DDoS attacks using artificial neural networks. Com- Peterson, L.; Rexford, J.; Shenker, S.; Turner, J.: OpenFlow:
mun. Comput. Inf. Sci. 172, 385–393 (2016) enabling innovation in campus networks. ACM SIGCOMM Com-
10. Hoque, N.; Bhattacharyya, D.; Kalita, J.: Botnet in DDoS attacks: put. Commun. Rev. 38(2), 69–74 (2008)
trends and challenges. IEEE Commun. Surv. Tutor. 99, 1–1 (2015) 31. Coughlin, M.: A survey of SDN security research. In: Future Net-
11. Arbor Networks Inc. http://www.arbornetworks.com works and Services (SDN4FNS), IEEE (2013)
12. Arbor networks detects largest ever DDoS attack in Q1 32. Kim, J.; Firoozjaei, M.D.; Jeong, J.P.; Kim, H.; Park, J.-S.:
2015 DDoS report. In: Arbor Networks (2015). http://www. SDN-based security services using interface to network security
arbornetworks.com/arbor-networks-detects-largest-ever-ddos- functions. In: 2015 International Conference on Information and
attack-in-q1-2015-ddos-report Communication Technology Convergence (ICTC), pp. 526–529.
13. Jain, S.; et al.: B4: experience with a globally-deployed software IEEE (2015)
defined WA. ACM SIGCOMM Comput. Commun. Rev. 43(4), 33. Yan, Q.; Yu, F.R.: Distributed denial of service attacks in software-
3–14 (2013) defined networking with cloud computing. IEEE Commun. Mag.
14. Technol, I.: Secure and Dependable SDNs, Feb 2016 (2015) 53(4), 52–59 (2015)
15. Shalimov, A.; Zuikov, D.; Zimarina, D.; Pashkov, V.; Smeliansky, 34. Giotis, K.; Argyropoulos, C.; Androulidakis, G.; Kalogeras, D.;
R.: Advanced study of SDN/openflow controllers. In: Proceed- Maglaris, V.: Combining OpenFlow and sFlow for an effective and
ings of the 9th Central & Eastern European Software Engineering scalable anomaly detection and mitigation mechanism on SDN
Conference in Russia on - CEE-SECR ’13 Oct pp. 1–6 (2013) environments. Comput. Netw. 62, 122–136 (2014)
16. Schehlmann, L.; Abt, S.; Baier, H.: Blessing or curse? Revisiting 35. Lee, W.; Xiang, D.: Information-theoretic measures for anomaly
security aspects of software-defined networking. In: Proceedings detection. In: Proceedings of the 2001 IEEE Symposium on Secu-
of the 10th International Conference on Network and Service rity and Privacy, S&P 2001, pp. 130–143. IEEE (2001)
Management, CNSM 2014, no. 1, pp. 382–387 (2015) 36. Gu, Y.; McCallum, A.; Towsley, D.: Detecting anomalies in net-
17. Kreutz, D.; Ramos, F.M.V.; Verissimo, P.: Towards secure and work traffic using maximum entropy estimation. In: Proceedings
dependable software-defined networks. In: Proceedings of the of the 5th ACM SIGCOMM conference on Internet Measurement,
second ACM SIGCOMM Workshop on Hot Topics in Software p. 32. USENIX Association (2005)
Defined Networking—HotSDN ’13, p. 55 (2013) 37. Bereziński, P.; Szpyrka, M.; Jasiul, B.; Mazur, M.: Network
18. Wang, B.; Zheng, Y.; Lou, W.; Hou, Y.T.: DDoS attack protection anomaly detection using parameterized entropy. In: Computer
in the era of cloud computing and software-defined networking. Information Systems and Industrial Management. Springer,
In: 2014 IEEE 22nd International Conference on Network Proto- Berlin (2014)
cols, pp. 624–629 (2014) 38. Nychis, G.; Sekar, V.; Andersen, D.G.; Kim, H.; Zhang, H.: An
19. Thapngam, T.; Yu, S.; Zhou, W.; Beliakov, G.: Discriminating empirical evaluation of entropy-based traffic anomaly detection.
DDoS attack traffic from flash crowd through packet arrival pat- In: Proceedings of the 8th ACM SIGCOMM Conference on Inter-
terns. In: 2011 IEEE Conference on Computer Communications net Measurement Conference—IMC ’08, p. 151 (2008)
Workshops, INFOCOM WKSHPS 2011, pp. 952–957 (2011) 39. Brauckhoff, D.; Tellenbach, B.; Wagner, A.; May, M.; Lakhina,
20. Xia, W.; Wen, Y.; Member, S.; Heng Foh, C.; Niyato, D.; Xie, H.: A.: Impact of packet sampling on anomaly detection metrics. In:
A survey on software-defined networking. IEEE Commun. Surv. Proceedings of the 6th ACM SIGCOMM conference on Internet
Tutor. 17(1), 27–51 (2015) measurement, pp. 159–164 (2006)
21. Liao, Q.; Li, H.; Kang, S.; Liu, C.: Application layer DDoS attack 40. Androulidakis, G.; Chatzigiannakis, V.; Papavassiliou, S.: Net-
detection using cluster with label based on sparse vector decom- work anomaly detection and classification via opportunistic
position and rhythm matching. Secur. Commun. Netw. 8(17), sampling. IEEE Netw. 23(1), 6–12 (2009)
3111–3120 (2015) 41. Wang, R.; Jia, Z.; Ju, L.: An entropy-based distributed DDoS
22. Stewart, J.M.: Network Security, Firewalls and VPNs. Jones & detection mechanism in software-defined networking. In: 2015
Bartlett Publishers (2013) IEEE Trustcom/BigDataSE/ISPA, pp. 310–317 (2015)
23. DDoS: website-crippling cyber-attacks to rise in 2016. BBC 42. Mehdi, S.,A.,S.; Khalid, J.; Khayam, S.,A.,S.: Revisiting traf-
News. http://www.bbc.com/news/technology-35376327 fic anomaly detection using software defined networking. In:
24. Q1 2016 Global DDoS Threat Landscape Report. Incapsula. Proceedings of the 14th International Conference on Recent
https://www.incapsula.com/blog/q1-2016-global-ddos-threat- Advances in Intrusion Detection, pp. 161–180 (2011)
landscape-report.html 43. Lakhina, A.; Crovella, M.; Diot, C.: Mining anomalies using traf-
25. Bawany, N.Z.; Shamsi, J.A.: Application layer DDoS attack fic feature distributions. ACM SIGCOMM Comput. Commun.
defense framework for smart city using SDN. In: Computer Rev. 35(4), 217 (2005)
Science, Computer Engineering, and Social Media (CSCESM) 44. sflow. http://www.sflow.com
(2016) 45. Fiadino, P.; Alconzo, A.,D.; Schiavone, M.; Casas, P.: Challeng-
26. Kreutz, D.; Ramos, F.M.V.; Verissimo, P.; Rothenberg, C.E.; ing entropy-based anomaly detection and diagnosis in cellular
Azodolmolky, S.; Uhlig, S.: Software-defined networking: a com- networks. In: Proceedings of the 2015 ACM Conference on Spe-
prehensive survey. Proc. IEEE 103(1), 14–76 (2015) cial Interest Group on Data Communication (2015)
27. Khondoker, R.; Zaalouk, A.; Marx, R.; Bayarou, K.: Feature- 46. Javed, M.; Ashfaq, A.B.; Shafiq, M.Z.; Khayam, S.A.: On the
based comparison and selection of Software Defined Networking Inefficient Use of Entropy for Anomaly Detection. Lecture Notes
(SDN) controllers. In: 2014 World Congress on Computer Appli- in Computer Science (including subseries Lecture Notes in Artifi-
cations and Information Systems (WCCAIS), pp. 1–7. IEEE cial Intelligence and Lecture Notes in Bioinformatics), vol. 5758
(2014) LNCS, no. c, pp. 369–370 (2009)
28. Berde, P.; Gerola, M.; Hart, J.; Higuchi, Y.; Kobayashi, M.; Koide, 47. Tsai, C.F.; Hsu, Y.F.; Lin, C.Y.; Lin, W.Y.: Intrusion detection by
T.; Lantz, B.; Snow, W.; Parulkar, G.; O’Connor, B.; Radoslavov, machine learning: a review. Expert Syst. Appl. 36(10), 11994–
P.: ONOS. In: Proceedings of the third workshop on Hot topics in 12000 (2009)
software defined networking—HotSDN ’14, pp. 1–6 (2014) 48. Sommer, R.; Paxson, V.: Outside the closed world: on using
29. Linux Foundation. http://www.opendaylight.org machine learning for network intrusion detection. In: 2010 IEEE
Symposium on Security and Privacy, pp. 305–316 (2010)

123
Arab J Sci Eng

49. Mukkamala, S.; Janoski, G.; Sung, A.: Intrusion detection using 67. Roesch, M.: Snort: lightweight intrusion detection for networks.
neural networks and support vector machines. In: Proceeding of In: LISA ’99: 13th Systems Administration Conference, pp. 229–
the 2002 International Joint Conference on Neural Networks, vols. 238 (1999)
1–3, pp. 1702–1707 (2002) 68. White, J.S.; Fitzsimmons, T.; Matthews, J.N.: Quantitative analy-
50. Kruegel, C.; Mutz, D.; Robertson, W.; Valeur, F.: Bayesian event sis of intrusion detection systems: Snort and Suricata. Proc. SPIE
classification for intrusion detection. In: Proceedings—Annual 8757, 875704 (2013)
Computer Security Applications Conference, ACSAC, pp. 14–23 69. Albin, E.; Rowe, N.C.: A realistic experimental comparison of
(2003) the Suricata and Snort intrusion-detection systems. In: 2012 26th
51. Kayacik, H.G.; Zincir-Heywood, A.N.; Heywood, M.I.: A hierar- International Conference on Advanced Information Network-
chical SOM-based intrusion detection system. Eng. Appl. Artif. ing and Applications Workshops (WAINA), pp. 122–127. IEEE
Intell. 20(4), 439–451 (2007) (2012)
52. Mabu, S.; Chen, C.; Lu, N.; Shimada, K.; Hirasawa, K.: An 70. Ali, S.T.; Sivaraman, V.; Radford, A.; Jha, S.: A survey of securing
intrusion-detection model based on fuzzy class-association-rule networks using software defined networking. IEEE Trans. Reliab.
mining using genetic network programming. IEEE Trans. Syst. 64(3), 1086–1097 (2015)
Man Cybern C Appl. Rev. 41(1), 130–139 (2011) 71. Chin, T.; Mountrouidou, X.; Li, X.; Xiong, K.: Selective packet
53. Abduvaliyev, A.; Pathan, A.-S.K.; Zhou, J.; Roman, R.; Wong, W.- inspection to detect DoS flooding using software defined net-
C.: On the vital areas of intrusion detection systems in wireless working (SDN). In: 2015 IEEE 35th International Conference
sensor networks. IEEE Commun. Surv. Tutor. 15(3), 1223–1237 on distributed Computing Systems Workshops (ICDCSW), pp.
(2013) 95–99. IEEE (2015)
54. Xu, Y.; Liu, Y.: DDoS attack detection under SDN context. In: 72. Xing, T.; Huang, D.; Xu, L.; Chung, C.J.; Khatkar, P.: Snort-
IEEE INFOCOM 2016—The 35th Annual IEEE International Flow: a OpenFlow-based intrusion prevention system in cloud
Conference on Computer Communications, pp.1–9. IEEE (2016) environment. In: Proceedings—2013 2nd GENI Research and
55. Braga, R.; Mota, E.; Passito, A.: Lightweight DDoS flooding Educational Experiment Workshop, GREE 2013, pp. 89–92
attack detection using NOX/OpenFlow. In: LCN ’10 Proceedings (2013)
of the 2010 IEEE 35th Conference on Local Computer Networks, 73. Von Ahn, L.; Blum, M.; Hopper, N.J.; Langford, J.: CAPTCHA:
pp. 408–415. IEEE, Washington (2010) using hard AI problems for security. In: International Conference
56. Dotcenko, S.; Vladyko, A.; Letenko, I.: A fuzzy logic-based on the Theory and Applications of Cryptographic Techniques, pp.
information security management for software-defined networks. 294–311. Springer, Berlin, Heidelberg (2003)
In: 16th International Conference on Advanced Communication 74. Piedrahita, A.F.M.; Rueda, S.; Mattos, D.M.F.; Duarte, O.C.M.B.:
Technology (ICACT), pp. 167–171. IEEE (2014) FlowFence: a denial of service defense system for software
57. Schechter, S.E.; Jung, J.; Berger, A.W.: Fast detection of scanning defined networking. In: 2015 Global Information Infrastructure
worm infections. In: International Workshop on Recent Advances and Networking Symposium (GIIS), Guadalajara, pp. 1–6. (2015)
in Intrusion Detection. Springer, Berlin, Heidelberg (2004) 75. Suciu, G.; Vulpe, A.; Halunga, S.; Fratu, O.; Todoran, G.; Suciu,
58. Chung, C.-J.; Khatkar, P.; Xing, T.; Lee, J.; Huang, D.: NICE: V.: Smart cities built on resilient cloud computing and secure
Network intrusion detection and countermeasure. IEEE Trans. internet of things. In: 2013 19th International Conference on Con-
Dependable Secure Comput. 10(4), 198–211 (2013) trol Systems and Computer Science (CSCS), pp. 513–518. IEEE
59. Dillon, C.; Berkelaar, M.: OpenFlow (D) DoS Mitigation. Techni- (2013)
cal Report (Feb 2014). http://www.delaat.net/rp/2013-2014/p42/ 76. Afaq, M.; Rehman, S.; Song, W.-C.: Large flows detection, mark-
report.pdf (2014) ing, and mitigation based on sFlow standard in SDN. J. Korea
60. Yen, T.-F.; Reiter, M.K.: Traffic aggregation for malware detec- Multimedia Soc. 18(2), 189–198 (2015)
tion. In: International Conference on Detection of Intrusions and 77. Sqalli, M.H.; Al-Haidari, F.; Salah, K.: Edos-shield-a two-steps
Malware, and Vulnerability Assessment, pp. 207–227. Springer, mitigation technique against edos attacks in cloud computing. In:
Berlin, Heidelberg (2008) 2011 Fourth IEEE International Conference on Utility and Cloud
61. Shin, S.; Porras, P.; Yegneswaran, V.; Fong, M.; Gu, G.; Tyson, Computing (UCC), pp. 49–56. IEEE (2011)
M.; Texas, A.; Station, C.; Park, M.: Fresco: modular composable 78. Salman, O.; Elhajj, I.H.; Kayssi, A.; Chehab, A.: SDN controllers:
security services for software-defined networks. In: Network and a comparative study. In: Proceedings of the 18th Mediterranean
Distributed System Security Symposium, pp. 1–16. (2013) Electrotechnical Conference (MELECON), pp. 1–6. IEEE (2016)
62. Gu, G.; Perdisci, R.; Zhang, J.; Lee, W.: BotMiner: clustering 79. Vissicchio, S.; Vanbever, L.; Bonaventure, O.: Opportunities and
analysis of network traffic for protocol- and structure-independent research challenges of hybrid software defined networks. ACM
Botnet detection. In: USENIX Security Symposium, vol. 5, no. 2, SIGCOMM Comput. Commun. Rev. 44(2), 70–75 (2014)
pp. 139–154 (2008) 80. Akyildiz, I.F.; Lee, A.; Wang, P.; Luo, M.; Chou, W.: A roadmap
63. Jin, R.; Wang, B.: Malware detection for mobile devices using for traffic engineering in software defined networks. Comput.
software-defined networking. In: GREE ’13 Proceedings of the Netw. 71, 1–30 (2014)
2013 Second GENI Research and Educational Experiment Work- 81. Akyildiz, I.F.; Lee, A.; Wang, P.; Luo, M.; Chou, W.: Research
shop, pp. 81–88. IEEE, Washington (2013) challenges for traffic engineering in software defined networks.
64. Twycross, J.; Williamson, M.M.: Implementing and testing a virus IEEE Netw. 30(3), 52–58 (2016)
throttle. In: Proceedings of the 11th USENIX Security Sympo- 82. Yu, S.; Zhou, W.; Jia, W.; Guo, S.; Xiang, Y.; Tang, F.: Discrim-
sium, pp. 285–294 (2003) inating DDoS attacks from flash crowds using flow correlation
65. Williamson, M.M.: Throttling viruses: restricting propagation to coefficient. IEEE Trans. Parallel Distrib. Syst. 23(6), 1073–1080
defeat malicious mobile code. In: Proceedings—18th Annual (2012)
Computer Security Applications Conference, pp. 61–68. IEEE 83. Lee, S.; Yoon, C.; Shin, S.: The smaller, the Shrewder: a simple
(2002) malicious application can kill an entire SDN environment. In: Pro-
66. Lim, S.; Ha, J.; Kim, H.; Kim, Y.; Yang, S.: A SDN-oriented DDoS ceedings of the 2016 ACM International Workshop on Security in
blocking scheme for botnet-based attacks. In: Sixth International Software Defined Networks & Network Function Virtualization,
Conference on Ubiquitous and Future Networks (ICUFN), pp. pp. 23–28. ACM (2016)
63–68. IEEE (2014)

123
Arab J Sci Eng

84. CCTV-based botnet used for DDoS attacks. https://www. 94. Hacker group ‘Anonymous’ claims credit for federal cyber
ddosattacks.net/a-massive-botnet-of-cctv-cameras-involved-in- attacks. http://ottawacitizen.com/news/politics/federal-comput
ferocious-ddos-attacks Accessed 04 July 2016 er-servers-cyber-attacked-clement. Accessed 04 Jul 2016
85. Sucuri, Inc. Delaware Corporation. https://sucuri.net 95. Musil, S.: Record-breaking DDoS attack in Europe hits 400 Gbps.
86. DDoS Attack on Bank of Greece Website https://www.hackread. CNET http://www.cnet.com/news/record-breaking-ddos-attack-
com/anonymous-ddos-attack-bank-greece-website-down. in-europe-hits-400gbps/
Accessed 04 July 2016 96. Cloudflare, Cloudflare Organization. https://www.cloudflare.
87. HSBC Internet Banking Services Down After DDoS Attack. com. Accessed 08 Aug 2015
http://www.telegraph.co.uk/finance/newsbysector/banksandfinan 97. Wong, F.; Tan, C.X.: A survey of trends in massive DDoS attacks
ce/12129411/HSBC-online-banking-service-crashes-again.html. and cloud-based mitigations. Int. J. Netw. Secur. Appl. (IJNSA)
Accessed 04 July 2016 6(3), 57–71 (2014).
88. HSBC Bank. www.hsbc.co.uk. Accessed 02 July 2016 98. Zargar, S.T.; Joshi, J.; Tipper, D.; Member, S.: A survey of defense
89. Irish Government Websites temporarily offline due to DDoS- mechanisms against distributed denial of service (DDoS). IEEE
attack. http://www.bbc.com/news/world-europe-35379817. Commun. Surv. Tutor. 15(4), 2046–2069 (2013)
Accessed 04 July 2016 99. Kaufman, C.; Perlman, R.; Sommerfeld, B.: DoS protection for
90. Laskar, S.; Mishra, D.: Qualified vector match and merge algo- UDP-based protocols. In: Proceedings of the 10th ACM Confer-
rithm (QVMMA) for DDoS prevention and mitigation. Procedia ence on Computer and communication security—CCS ’03 p. 2,
Comput. Sci. 79, 41–52 (2016) (2003)
91. Web Attack Knocks BBC Websites Offline. http://www.bbc.com/ 100. Peng, T.; Leckie, C.; Ramamohanarao, K.: Survey of network-
news/technology-35204915. Accessed 04 July 2016 based defense mechanisms countering the DoS and DDoS prob-
92. Thai Government Websites hit by denial-of-service attack. http:// lems. ACM Comput. Surv. 39(1), 3-es (2007)
www.bbc.com/news/world-asia-34409343. Accessed 04 July 101. Czyz, J.; Kallitsis, M.; Papadopoulos, C.; Bailey, M.: Taming the
2016 800 Pound Gorilla: the rise and decline of NTP DDoS attacks. In:
93. Hack attack leaves 1,400 airline passengers grounded. IMC, pp. 435–448 (2014)
http://www.cnbc.com/2015/06/22/hack-attack-leaves-1400-pass
engers-of-polish-airline-lot-grounded.html. Accessed 04 July
2016

123

View publication stats

You might also like