NetworkSurveillanceSystems:ModelsandApproaches

by
DanielPestov

AThesissubmittedtotheFaculty
inpartialfulfillment
oftherequirementsforthe
BACHELOROFARTS

Accepted

PaulShields,ThesisAdvisor

MichaelBergman,MajorAdvisor

SimonsRockCollege
GreatBarrington,Massachusetts
2008
TableofContents

Introduction.......................................................................................................................................................................1
1AttackersInventory.................................................................................................................................................7
1.1 IPSpoofing......................................................................................................................................................7
1.2 TCPSessionHijacking................................................................................................................................8
1.3 DenialofService...........................................................................................................................................9
1.3.1 BandwidthDepletion......................................................................................................................11
1.3.2 ResourceDepletion..........................................................................................................................12
1.3.3 DoSResponse.....................................................................................................................................13
1.4 NetworkProbes..........................................................................................................................................14
1.5 Shellcode........................................................................................................................................................16
1.6 Polymorphicshellcode.............................................................................................................................18
2SignatureDetection................................................................................................................................................22
2.1 Approaches...................................................................................................................................................23
2.1.1 StringMatching.................................................................................................................................23
2.1.2 StateModeling...................................................................................................................................23
2.1.3 ExpertSystem....................................................................................................................................24
2.1.4 ColoredPetriNets............................................................................................................................25
2.1.5 RuleBasedSystems.........................................................................................................................27
3AnomalyDetection..................................................................................................................................................29
3.1 FirstIntrusionDetectionModel...........................................................................................................32
3.2 SourcesofAuditData................................................................................................................................34
3.3 ProtocolAnomalies...................................................................................................................................36
3.4 StatisticalAnomalies.................................................................................................................................38
3.5 Approaches...................................................................................................................................................38
3.5.1 ProtocolStateModeling.................................................................................................................39
3.5.2 SignalProcessingofMIBVariables...........................................................................................42
3.5.3 DataMiningUsingClusters...........................................................................................................43
3.5.4 DetectingSYNFloods......................................................................................................................47
3.6 HybridIntrusionDetection....................................................................................................................48
4Response.....................................................................................................................................................................49
5NIDSforDistributedNetworks..........................................................................................................................53
5.1 EMERALD......................................................................................................................................................53
5.1.1 EMERALDMonitorArchitecture................................................................................................55
5.2 GrIDS................................................................................................................................................................56
6EfficiencyEvaluation..............................................................................................................................................59
6.1 Benchmarking..............................................................................................................................................61
6.1.1 CaseStudy:BenchmarkingBasedonDataEntropy...........................................................61
7RelatedWorks...........................................................................................................................................................65
7.1 Honeypots.....................................................................................................................................................65
7.1.1 Honeynets............................................................................................................................................68
7.1.2 DynamicHoneypots.........................................................................................................................71
7.2 Honeycomb...................................................................................................................................................72
7.3 IdentifyingtheSourceofDoSAttacks................................................................................................73
Conclusion........................................................................................................................................................................75
References........................................................................................................................................................................76

ListofFigures

Figure1:IDSArchitecture................................................................................................................................................................5
Figure2:Smurf....................................................................................................................................................................................12
Figure3:StatebasedIntrusionscenario................................................................................................................................24
Figure4:CPA(representingpartialorderingofevents)..................................................................................................26
Figure5:TCPStateMachine.........................................................................................................................................................40
Figure6:Anomalydetectionusingclusters...........................................................................................................................46
Figure7:HybridIDS.........................................................................................................................................................................48
Figure8:Wormactivitygraph.....................................................................................................................................................57
Figure9:Reducingdepartmentsintoasinglenode...........................................................................................................58
Figure10:Honeynetarchitecture..............................................................................................................................................69

Abstract:

This paper presents a technical overview of network intrusion

detection, with a particular emphasis on anomaly detection
systems. At the core of the discussion is a comparative analysis of
two complementary paradigms of network surveillance: signature
based detection and anomalybased detection. Signaturebased
detection systems analyze network traffic for patterns of known
intrusive activity while anomaly detection systems look for
deviations from normal system behavior. The discussion focuses on
data models, approaches and response methodologies. We also
introduce architectures of several commercial intrusion detection
engines as well as complementary technologies and areas of
researchthatcharacterizepowerfulyetnontraditionalapproaches
tonetworksecurity.


1

Introduction
Bue to eveiincieasing ieliance on netwoikbaseu electionic communication
technology theie is a giowing uemanu foi piotecting the infiastiuctuie anu uata which
iesiues on the netwoik. Netwoik secuiity must meet societys giowing uepenuency on
the Inteinet foi ecommeice, banking, uefense, healthcaie, communications, eneigy
management, anu othei ciitical applications which have become an inuispensible pait
of uaily living. Existing secuiity safeguaius on netwoiks anu hosts usually begin with
authentication schemes such as access contiol mechanisms (e.g. passwoius, fiiewalls,
anu enciyption schemes). Bowevei, such measuies pioviue only a limiteu layei of
piotection of the system since these authentication mechanisms can be eithei evaueu oi
compiomiseu. Nisconfiguieu fiiewalls can be bypasseu anu passwoius might be stolen
oi guesseu. Attacks against secuie systems anu uata can oiiginate fiom malicious useis
oi systems fiom outsiue oi insiue an oiganization.
Accoiuing to CSIFBI Computei Ciime anu Secuiity Suivey the total cybeiciime
losses in 2uuS ieacheu $1Su.1 million; the majoiity of these losses weie uue to viiuses,
unauthoiizeu access to computei systems anu theft of infoimation |S2j. Accoiuing to
CERT (Computei Emeigency Response Team) |S8j, the total numbei of catalogeu
vulneiabilities in 2uu7 was 7,2S6, while only 171 weie inuexeu in 199S. vulneiabilities
aie weaknesses in piotocols, netwoik seivices, opeiating systems anu neaily all types
of netwoik infiastiuctuie, causeu by bugs, misconfiguiations anu othei flaws of vaiious
kinus. Attackeis aie malicious useis who exploit vulneiabilities in oiuei to gain
unauthoiizeu access into systems anu netwoiks, steal confiuential infoimation, uisiupt
opeiation of seivices anu ueny legitimate useis access to iesouices. Fixing all


2

vulneiabilities is not a viable solution uue to time, knowleuge ueficiency, anu iesouice
constiaints. Netwoiks aie uynamic systems constantly changing theii topology, auuing
new seivices, new piotocols, anu new systems ovei time. Fuitheimoie, outuateu assets
cannot be easily ieplaceu by new stanuaius uue to high inteiopeiability anu tight
integiation that peivaues infoimation systems. In many financial oiganizations entiie
uepaitments exist only to pioviue suppoit foi long outuateu uevices which aie neeueu
by an oiganization.
To avoiu the aims iace between vulneiability uiscoveiy anu patching local
systems, secuiity piofessionals anu ieseaicheis have focuseu ieseaich on attack
uetection anu iecoveiy. 0ne uetection stiategy is to assume attacks will occui
eventually anu the goal woulu be to uesign a system to become awaie of such attacks. In
the event of an attack, the system will aleit iesponsible paities anuoi attempt to iepaii
the uamage causeu by it in some kinu of oiganizeu fiamewoik.
Secuiity systems that pioviue intiusion uetection capabilities aie calleu
Intrusion Detection Systems (IBS). An intrusion is uefineu as an attempt by a usei of a
system to peifoim an action heshe was not legally alloweu to peifoim |11j. Noie
geneially, an intiusion is any set of actions that attempts to compiomise the integiity,
confiuentiality oi availability of a potentially valuable iesouice, oi to bypass the
secuiity mechanisms of a computei system oi a netwoik with the goal of negatively
affecting the oiganizations inteiests.
Confiuentiality, availability anu integiity aie the thiee coie piinciples that
motivate the uevelopment anu ueployment of intiusion uetection systems.
Confidentiality iefeis to effoits to conceal anu ueny access to infoimation anu iesouices


S

to unauthoiizeu useis. Acciuental oi intentional uisclosuie of sensitive infoimation oi
peisonal uata has a potential to uestioy ieputation of fiims anu businesses, unueimine
business tiansactions, even put peoples lives in uangei. Inuustiial fiims, goveinment
anu militaiy agencies, health caie pioviueis take many steps to ensuie that iecoius
containing peisonal uata about theii customeis, employeis, patients anu opeiatives is
shaieu only among authoiizeu peisons anu oiganizations. Regulatoiy agencies pass
legislations, such as Saibanes0xley Act (S0X) anu Bealth Insuiance Poitability anu
Accountability Act (BIPAA), iequiiing oiganizations to institute appiopiiate stiategies
of piotecting piivacy of peisonal anu sensitive infoimation |S1j. Inteiception of
infoimation by an unauthoiizeu paity is calleu snooping. It may involve listening to
communication, biowsing thiough files, etc.
Integrity iefeis to effoits to pievent impiopei anu unauthoiizeu mouification to
infoimation oi iesouices. Loss of integiity also iesults when uata is eithei acciuentally
oi maliciously ueleteu, oi when new uata is cieateu without appioval by an authoiizing
paity. Integiity ensuies that infoimation pioviueu to public oi stoieu foi inteinal use is
authentic, tiustwoithy anu uptouate. We note that sometimes a loss of integiity can be
attiibuteu to incompetence iathei than malicious intent, but the enu iesults aie the
same.
Availability means that infoimation oi iesouices can be useu by those foi whom
it is intenueu. Availability is compiomiseu when someone uelibeiately attempts to ueny
access to uata oi seivices pioviueu by a system by eithei making it completely
unavailable oi unueimining some subset of its functionality. Such attacks aie calleu the
DenialofService attacks anu aie often veiy uifficult to uetect.


4

Intiusion uetection systems monitoi a specific taiget oi a list of taigets. Each
taiget is a uynamic system that geneiates events of inteiest. It coulu be a woikstation, a
web seivei, a uatabase, an FTP seivice, a iouting uevice, an entiie netwoik, etc. Events
aie auuitable changes in the state of the system. An event can iepiesent a single action
by a usei oi a system, oi a seiies of actions, that when peifoimeu in sequence, piouuce
a single obseivable iecoiu. Some examples of events incluue tiansmission oi
inteiception of a netwoik packet, usei iunning a commanu, uatabase tiansaction, login
attempt, etc. Events monitoieu by an intiusion uetection system aie calleu auditevents.
To uetect intiusion is to finu eviuence that it is taking place. The inciuence of intiusion
is eithei ieflecteu in the auuit tiail it geneiates oi the subsequent state of the system.
0nce auuit uata has been collecteu, intiusion uetection system employs one of
two possible analysis methouologies: signatuiebaseu oi anomalybaseu. Signature
baseddetection, also known as misusedetection, uses knowleuge accumulateu about the
intiusive piocess anu looks foi specific tiaces it ought to leave in the taiget system. In
othei woius signatuiebaseu systems uefine illegal behavioi anu each signatuie
iepiesents a specific attack scenaiio. As a iesult such systems achieve high uegiee of
accuiacy in uetecting even the most subtle attacks. The majoi shoitcoming of this
appioach is that it is limiteu to only known attacks.
Anomalybased detection systems (ABS) on the othei hanu, assume no piioi
knowleuge about the attacks they aie expecteu to uetect. Insteau, they iuentify events
that appeai to be anomalous with iespect to noimal system behavioi. The piincipal
assumption confiimeu by piactical application is that abnoimal behavioi is a sign of
intiusion oi system malfunction. The main auvantage of anomaly uetection ovei


S

Det ect ion Engine
Analysis Engine
Response Engine
Figure 1: IBS Aichitectuie
Auditeventstream
Intrusionreport
signatuie uetection is the ability to iuentify new anu unfoieseen attacks. Bowevei, not
eveiy anomaly is tiiggeieu by intiusion. Theiefoie, anomalybaseu uetection systems
aie known foi geneiating high numbeis of false alaims.
Intiusion uetection systems aie also classifieu in teims of theii monitoiing
scope. Hostbased intrusion detection systems (BIBS) monitoi the behavioi of the
inuiviuual machine on which they aie ueployeu. Such systems typically analyze file
system activity, system call invocations, application logs, etc. Networkbased intrusion
detection systems (NIBS) iuentify intiusions by examining netwoik tiaffic anu
monitoiing multiple hosts as well as othei netwoiking uevices such as iouteis,
switches, etc. Each inuiviuual netwoik is a collection of computeis, useis anu seivices
with specific behavioial patteins that aie ieflecteu in the geneiateu netwoik tiaffic. The
puipose of netwoikbaseu intiusion uetection is to scan netwoik tiaffic foi events that
chaiacteiize a potential violation of secuiity, also known as a threat. An actual violation
of secuiity is calleu an attack. Not all thieats manifest themselves as attacks; some aie
false alaims, yet otheis aie causeu by malfunctioning netwoik uevices, etc.
A typical highlevel aichitectuie of
intiusion uetection system is illustiateu in figuie
1. Detection engine collects auuit uata. Analysis
engine analyzes auuit uata anu geneiates an
intiusion iepoit foi eveiy iuentifieu intiusion.
Response engine uevises anu implements
couteimeasuies.


6

This thesis is an oveiview of netwoik intiusion uetection systems, the
motivation behinu them, theii uesign anu aichitectuie. The focus will be placeu on
anomaly uetection systems. The iest of the papei is oiganizeu as follows. Chaptei 1
intiouuces the types of tools anu techniques useu by intiuueis touay to accomplish
theii goals. Chaptei 2 seives as a technical intiouuction into signatuiebaseu intiusion
uetection systems. Chaptei S pioviues a moie uetaileu suivey of anomaly uetection. In
chaptei 4, we uiscuss vaiious iesponse methouologies. Chaptei S will intiouuce
scalable netwoik suiveillance aichitectuies. In chaptei 6 we uiscuss how to measuie
the peifoimance of intiusion uetection systems. Chaptei 7 will intiouuce vaiious
complementaiy solutions anu ielateu aieas of ieseaich.



7

Chapter1
AttackersInventory

Because touays computei systems aie composeu of many inuepenuent mouules which
act togethei to achieve the goals of an oiganization, the iisk can take on many facets.
Attacks anu compiomises can come fiom many uiffeient vectois, some of which have
been aiounu foi a long time anu otheis newly inventeu. Sometimes they take the foim
of vulneiabilities of the local softwaie system. 0thei times they take auvantage of
misconfiguiations of the netwoik oi ambiguous secuiity settings when multiple
systems inteiact. We now suivey some populai attacks

1.1 IPSpoofing
IP Spoofing is a technique often useu by hackeis that involves masking souice IP
auuiess by manipulating IP heauei of outgoing packets. Routing uevices use uestination
auuiess fielu to ioute packets acioss the netwoik anu ignoie souice IP, which is only
useu by the uestination machine when it neeus to senu a iesponse. IP spoofing is useful
foi concealing iuentity, pietenuing to be a tiusteu host anu caiiying out many types of
attacks such as uenial of seivice, session hijacking, etc.
Few basic piecautionaiy measuies can be taken to limit the iisk of IP spoofing
on a netwoik. Since most spoofingielateu attacks oiiginate fiom outsiue the taiget
netwoik anu use IP auuiesses belonging to hosts insiue the netwoik, the most


8

ieasonable solution is to ueny access to any incoming packet whose souice auuiess
belongs to the oiganizations inteinal IP iange. This technique is calleu ingressfiltering
|SSj. Egress filtering is the opposite stiategy; outgoing packets aie blockeu if theii
souice IP auuiess belongs to the host outsiue youi netwoik. This pievents someone on
the netwoik fiom senuing spoofeu tiaffic to the inteinet. Foi example in the event of a
machine being compiomiseu, an oiganization wants to ensuie theii iesouices aie not
useu against an outsiue netwoik. Enciyption anu authentication mechanisms can also
be useu to limit most spoofing thieats. Nany cuiient shoitcomings aie auuiesseu in the
upcoming ievision of IP piotocol (IPv6) once it becomes wiuely auopteu anu
establisheu.

1.2 TCPSessionHijacking
TCP Session hijacking, also known as the maninthemiddle attack, is an attack that
involves the inteiception of an alieauy establisheu anu authenticateu TCP session
between two communicating paities (client anu seivei) by an offenuing paity.
Fiist, the attackei inteicepts one of the packets exchangeu between the two
machines anu extiacts clients IP auuiess, poit anu next sequence numbei expecteu by
the seivei. Next, they foige theii own packet by spoofing clients IP anu poit numbei as
well as inseiting the sequence numbei the seivei expects to ieceive fiom the client on
the next tiansmission. In oiuei to assume clients iuentity, the attackei senus this
packet to the seivei. The seivei will assume that they aie still communicating with the
client since in some cases they iuentify client only by IP auuiess anu poit numbei.


9

Theie is still one pioblem: the client is still an active paiticipant in this TCP
session, so how can the attackei guaiantee that the seivei only accepts theii own
packets anu not the clients. The solution lies in the fact that TCP piotocol uniquely
iuentifies each incoming packet within the same session by its sequence numbei. When
two packets aiiive having the same sequence numbei, the lattei is always uisiegaiueu
anu uioppeu. The attackei can uesynchionize clients communication with the seivei
by always senuing his own packets aheau of clients packets with the same sequence
numbei.
When the TCP session is successfully inteicepteu, neithei the client noi the
seivei is awaie that theie is an inteimeuiaiy piesent. Since the seivei tieats the client
as a tiusteu host, the attackei who assumes clients iuentity can fool the seivei into
uisclosing piivate infoimation oi iunning commanus on his behalf.
Anothei vaiiation of this attack is to foige netwoik iuentifieis so that the client
thinks the malicious machine is the uestination machine. Foi example, if a client is
tiying to connect to something.com, the attackei will make believe they aie
something.com. When the client attempts to connect to something.com, the attackei
will connect to something.com on theii behalf, anu pass back the intenueu infoimation.
At the same time the attackei is fiee to both monitoi anu mouify any infoimation. This
is uisastious if the client is connecting to a bank account oi othei sensitive uata souices.

1.3 DenialofService
Denial of service attack (DoS) is an attempt to compiomise the effective opeiation oi
availability of a netwoik iesouice oi seivice to its intenueu useis. Theie aie two classes


1u

of uenial of seivice attacks: banuwiuth uepletion anu iesouice uepletion |S4j.
Bandwidthdepletion attacks, also known as flooding, satuiate the taiget netwoik with a
laige amount of tiaffic, using up all available banuwiuth anu pieventing legitimate
iequests fiom ieaching theii uestination host. Resource depletion attacks tiiggei
excessive consumption of computational iesouices (CP0, memoiy, uisk space) of the
taiget seivei, unueimining its ability to piocess usei iequests.
The victim of a BoS attack can be any computei system which inteiacts with the
outsiue. Examples incluue:
Netwoik seivei oi client
Inteinet seivice pioviuei
Lowlevel netwoik uevice (ioutei, switch, oi pioxy)
Inuiviuual netwoik link oi an entiie netwoik
v0IP phone
Inuiviuual inteinet usei oi a company conuucting its business using the
Inteinet.
0ne lowlevel example is attacking necessaiy components of netwoik piotocol. Foi
example the shutuown of the entiie netwoik can be achieveu by simply uisabling oi
making inaccessible a Bomain Name Lookup (BNS) system. This can be accomplisheu
by senuing small BNS queiies containing spoofeu IP auuiess of the taiget computei to
each BNS seivei on the netwoik. Laige iesponses aie sent to the taiget, iesulting in link
congestion anu potential uenial of inteinet connectivity. By senuing many BNS queiies,
the netwoik link to BNS seivei may become flooueu with excessive tiaffic anu
consequently the BNS iesponse time will be uegiaueu, peihaps to the point of making


11

the entiie netwoik unusable. An attackei can geneiate tiaffic suiges using his own host
oi by using othei compiomiseu machines on his behalf. Such cooiuinateu laigescale
BoS attack is calleu DistributedDenialofService (BBoS). Each seconuaiy system that the
attackei contiols anu uses to wage his attack is calleu an agent.

1.3.1 BandwidthDepletion
ICNP is a netwoik layei piotocol that facilitates the tiansmission of status anu eiioi
messages between computeis on a netwoik. Pinging is a piocess of uiscoveiing live
hosts on a netwoik. It is accomplisheu by senuing ICNP echo iequest packets to the
taiget host anu listening foi ICNP echo iesponse ieplies |9j. ICMPflood iesults when
multiple ICNP pings oveiloau a system by foicing it to ieuiiect all its available
iesouices to issuing echo iesponses, often to the point that legitimate netwoik tiaffic
can no longei be seiviceu. In oiuei to conceal his iuentity (oi the iuentity of agents in
the event of BBoS), the attackei can supply spoofeu IP auuiess to all echo iequests anu
uiiect all echo iesponses to anothei system.
A smurf is a wellknown vaiiant of ICNP floou attack with an amplifieu
uamaging effect. A smurf piogiam geneiates ICNP echo iequest packet with spoofeu
souice IP auuiess anu senus it to a bioaucast auuiess (See figuie 2). A bioaucast
auuiess is a special IP auuiess that allows packets to be sent to all machines on a given
subnet iathei than a specific host. All the echo iesponses aie sent back to the spoofeu
auuiess (a victim host). If enough pings aie geneiateu anu bioaucasteu this way, the
entiie netwoik coulu be biought to a state that it can no longei iesponu to a legitimate
tiaffic.


12


1.3.2 ResourceDepletion
SYNflood, also known as Neptune, is a type of BoS attack that exploits TCP thieeway
hanushake |19j. The usual pioceuuie foi initiating a TCP connection with a seivei is by
senuing a SYN packet to which a seivei iesponus with a SYNACK packet. The
connection is completeu when the client senus a final acknowleugment (ACK packet).
The pioblem, howevei, is that most seiveis uont wait foi a final ACK; they allocate
iesouices as soon as they ieceive a SYN. An attackei coulu exploit this fact by senuing a
seiies of SYN packets anu cieating many halfopeneu connections that consume
iesouices. Iule TCP connections eventually expiie, but if the attackei can match the iate
of expiiation with the iate of floouing, he coulu inuefinitely suspenu the opeiation of
the seivei, effectively uenying seivice to all othei clients who wish to access its
iesouices oi use its seivices.
LAND attack is a type of BoS attack that involves senuing a specially ciafteu
spoofeu SYN packet to a taiget machine wheie uestination IPpoit matches souice
IPpoit. This causes taiget to ieply to itself continually, iesulting in a system lockup.
attacker
router
(amplifier)
agentsystems
victim
Figure 2: Smuif


1S

1.3.3 DoSResponse
0nce BoS attack has been uetecteu by NIBS, the alaim is geneiateu anu netwoik
auministiatois aie notifieu. Two couises of action can be taken at this point: (1) iestoie
noimal netwoik opeiation by alleviating the tiaffic loau oi (2) iuentify the souice of the
attack. Theie aie seveial ways to countei the effects of BoS floouing:
Throttling: ueploy a fiiewall to limit the incoming banuwiuth anu filtei out
specific packet types useu in the attack (ICNP, SYN). Foi example, to mitigate
SYN floou, a fiiewall can monitoi the numbei of SYNs to a seivei ovei a
specific time inteival anu not allow the numbei to exceeu a configuieu
thiesholu. The siue effect of this technique is that legitimate tiaffic will also
get uioppeu.
SYN Proxy: Anothei piotection against SYN floou is to ueploy a pioxy that
inteicepts TCP connection iequests. Foi any halfopeneu connection, the
pioxy woulu automatically issue an RST (ieset) packet, which will close the
TCP connection anu fiee up seiveis iesouices.
Loadbalancing: in the event of the floou, ieuiiect tiaffic to multiple machines
as a way of ieuucing link congestion anu fieeing up seivei iesouices. 0se
some foim of scheuuling to assign new connection iequests to uiffeient
seiveis. New seiveis aie auueu oi iemoveu accoiuing to the iate of floouing.



14

1.4 NetworkProbes
Nany attacks scenaiios iequiie attackeis to gain infoimation about the taiget netwoik
as a pieliminaiy to the actual attack. Probes oi scanners aie piogiams anu techniques
that systematically scan the host oi an entiie netwoik foi vulneiabilities that can be
exploiteu. Scanning facilitates the uetection of live hosts, open poits, active TCP0BP
seivices anu native opeiating system. Ping sweeping ueteimines if a given IP auuiess
belongs to a live host. Port scanning iuentifies listening poits on a taiget system anu
seivices iunning on them. Netwoik seivices listen foi connections at wellknown
poits. Theie aie 6SSSS uistinct anu usable poits. Fingerprinting is a piocess of
uetecting host opeiating systems.
ICMPsweep is a simple ping sweep that utilizes ICNP (Inteinet Contiol Nessage
Piotocol) |1j. ICNP sweep consists of ICNP ECB0 iequest packets sent to multiple
hosts on the netwoik, usually in a shoit peiiou of time. ICNP ECB0 ieply packet is
issueu when a specifieu IP auuiess belongs to a live host. A single ICNP ECB0 iequest
coulu also be sent to a bioaucast auuiess which multiplies the packet anu senus it to
eveiy single host on a given subnet.
TCP sweep is anothei common ping sweep. It is accomplisheu by senuing TCP
SYN packet to common TCP poits |1j. If the seivei is live anu the poit is set to listen, the
seivice iunning on it will acknowleuge the connection iequest. If the poit is not active,
the seivei will senu RESET segment in oiuei to teai uown the connection. Receiving
any iesponse is a goou inuication that IP auuiess is assigneu to a live host.
SYNscan is a poit scanning technique accomplisheu by senuing a SYN packet to
eveiy poit on a system |1j. Reply in a foim of SYNACK packet inuicates that the poit is


1S

open, while the ieception of RSTACK packet inuicates that the given poit is closeu. As
soon as the connection is acknowleugeu by the seivei, the attackei immeuiately teais it
uown in oiuei to avoiu logging mechanisms.
Sometimes scanning is accomplisheu by senuing unexpecteu oi invaliu packet to
the taiget system. In such cases TCP specification iequiies taiget opeiating system to
iesponu with RESET packet foi poits that aie closeu, anu senu nothing if poit is open.
Common piobe packets aie FIN, SYNACK, XNAS (all flags set) anu N0LL (none of the
flags aie set).
Nmap (Netwoik Nappei) is a famous utility that pioviues auvanceu poit
scanning capabilities. Foi each given poit, it ietuins its state (open, closeu oi filteieu)
anu seivice iunning on it (ftp, http, tcpmux, etc). Filteieu poit inuicates a piesence of a
fiiewall, which piohibits access to ceitain open poits. Noimally a host woulu iesponu
with a SYNACK in iesponse to SYN packet if the poit is open anu RST if the poit is
closeu. Fiiewalls make it impossible to uiffeientiate between filteieu anu closeu poits
by blocking both types of iesponses. Theie is howevei a walkaiounu calleu firewalking.
The iuea is to set TTL (time to live) option on a TCPIP packet to the numbei of hops it
takes to ieach the taiget host. If fiiewall allows incoming tiaffic (open poit), then by the
time the packet ieaches the taiget system, its TTL will ieach u anu seivei will senu a
Time Exceeueu ICNP contiol packet back to the attackei. If the poit is closeu, the
fiiewall will simply uiop the uatagiam anu the attackei will not ieceive any iesponse.
Stack fingerprinting is a technique of uiscoveiing iemote opeiating system |1j.
0peiating system venuois uo not follow official TCPIP specifications exactly as
publisheu in RFCs. This allows hackeis to senu special TCP packets to the taiget anu


16

ueteimine 0S iunning on it by obseiving the iesponses, which will vaiy uepenuing on
the 0S. We piesent two examples |1uj:
FIN piobe packets sent to an open poit aie supposeu to be ignoieu accoiuing
to official guiuelines. Some systems, howevei iesponu with a RESET packet
(Winuows, BSBI, CISC0).
0peiating systems can be uiffeientiateu baseu on which TCP options they
implement. When a host iesponus to a queiy, the ieply will only contain
those options which aie suppoiteu. By senuing a TCP queiy with all options
set, the attackei can ueteimine opeiating system by obseiving which options
weie set in the ieply.
So fai we have uiscusseu active fingeipiinting baseu on piobing. Theie is also passive
fingerprinting baseu on sniffing |24j. 0nlike piobes which senu malfoimeu packets in
oiuei to illicit a specific iesponse fiom a iemote host, a sniffei captuies outgoing
packets anu passively examines theii heaueis. p0f is a populai passive fingeipiinting
tool that compaies captuieu packets against a uatabase of signatuies. Each signatuie
encoues values of packet heauei fielus that aie unique foi a TCPIP stack of a given 0S.

1.5 Shellcode
Shellcode is a small segment of coue, usually wiitten in assembly language, which acts
as a payloau injecteu into a computei system via vulneiability in softwaie with netwoik
inteiface. 0nce executeu, it opens a commanuline inteiface (ioot shell) on a taiget
system anu iuns commanus on behalf of the attackei. The instiuctions will often uisable


17

secuiity mechanisms, open a backuooi to allow iemote access, oi facilitate the tiansfei
anu installation of auuitional malicious coue to fuithei compiomise the system.
Buffer overflow is a conuition that occuis when an application ieceives moie
input that it can stoie insiue its fixeusizeu input buffei. This pioblem has been aiounu
foi ovei thiity yeais, anu still plagues computei systems anu seivices. A common
example: many piogiams wiitten in C misuse libiaiy functions, such as strcpy anu
sprintf, anu uo not check the length of theii aiguments. Buffei oveiflow vulneiability
iesults when a piogiam accepts input stiing fiom a usei anu copies it into a local buffei
using these calls without explicit bounu checking. If the attackei pioviues a stiing laigei
than the allocateu buffei, the aujacent memoiy will be oveiwiitten. If the supplieu
stiing contains bytes that aie actual native CP0 instiuctions, the attackei can take
contiol of the piogiam anu execute aibitiaiy coue. Fuitheimoie, if the piogiam is
iunning in piivilegeu moue, the attackeis coue inheiits those piivileges.
All local buffeis aie stoieu in a poition of piogiam memoiy calleu the stack, oi
execution stack. The stack is a uynamic uata stiuctuie that stoies infoimation
associateu with function calls. Each time a function is calleu, a new activation iecoiu is
allocateu on a stack that stoies, among othei things, all iequiieu buffeis anu ietuin
auuiess memoiy pointei to the next instiuction that neeus to be executeu on function
exit. In oiuei to guaiantee the execution of injecteu coue, the attackei also neeus to
oveiwiite the ietuin auuiess on the stack to point to the malicious coue. This also
iequiies figuiing out at what auuiess the buffei, anu thus the injecteu coue itself, will
be. To inciease the chances of pointing to the coiiect auuiess, attackeis often pau the
fiont of the oveiflow stiing with a sequence of N0P instiuctions that peifoim no


18

opeiation, calleu NOPsled |2uj. If the ietuin auuiess points anywheie within N0P sleu,
shellcoue will be successfully executeu. A typical shellcoue exploit consists of thiee
paits: a set of N0P instiuctions, payloau (injecteu coue) anu ietuin auuiess. It can be
conceptually iepiesenteu as:
|N0P SLEBj|PAYL0ABj|RET0RN ABBRESSj

1.6 Polymorphicshellcode
Since signatuiebaseu uetection systems iuentify attack instances by matching specific
patteins encoueu in theii knowleuge base against those obseiveu in live tiaffic,
signatuiebaseu uetection can be bypasseu by geneiating malicious coue which is
functionally equivalent to the oiiginal, but syntactically alteieu. Attacks that have been
mouifieu foi the puipose of being evaueu by NIBS aie calleu polymorphicattacks.
Tiauitionally the uetection of most wellknown shellcoue injection attacks was
caiiieu out by simple stiingbaseu signatuie matching techniques. In polymoiphic
shellcoue attacks |2uj, the attackei can chose an aibitiaiy enciyption algoiithm to
enciypt attack coue anu effectively geneiate a semantically equivalent veision of the
oiiginal buffei oveiflow exploit. The oiiginal signatuie woulu no longei be useful.
Theiefoie, the new exploit woulu no longei be uetectable by signatuiebaseu intiusion
uetection systems, at least until the new signatuie itself is auueu to the knowleuge base.
ADMmutate IBS evasion tool is a wellknown application that systematically
geneiates polymoiphic shellcoue. The tool takes as input a buffei oveiflow exploit, anu
puts it thiough a mutation piocess in oiuei to piouuce thousanus of functionally


19

equivalent veisions of the same uata. Each poition of shellcoue exploit is uisguiseu in a
way that ciicumvents signatuie uetection but pieseives oveiall opeiational semantics.
|N0P SLEBj Nany signatuiebaseu NIBS sensois peifoim pattein matching
to uetect N0P stiings useu in known exploits. Attackeis, howevei, founu
ways to apply polymoiphism to hiue N0P sleu. It tuins out theie aie SS
singlebyte instiuctions on Intel aichitectuie that can seive as opeiationally
equivalent alteinatives foi N0P instiuction. This fact allows polymoiphic
engines, like ABNmutate, to ianuomly geneiate SS
n
unique nbyte sequences
of noeffect instiuctions |2uj.
|RET0RN ABBRESSj Since location of the stack anu, consequently, the ietuin
auuiess is unlikely to change, it woulu seem ieasonable to geneiate
signatuies that specifically taiget these wellknown ietuin auuiess values to
uetect coue injection attacks. Bowevei, as it tuins out, ietuin auuiess can
also be uisguiseu using polymoiphic techniques. In paiticulai, one coulu
mouulate the least significant byte of the auuiess fielu anu cause the contiol
flow to jump into aibitiaiy position in the N0P sleu |Suj. The exploit woulu
still be executeu.
|PAYL0ABj ADMmutate uses simple X0R enciyption with a ianuom key to
uisguise payloau |Suj. Polymoiphism can also be achieveu by using multiple
iounus of cipheiing, ianuomizing theii oiuei anu by using uiffeient keys foi
each. The uecouing component is also placeu insiue shellcoue. The execution
flow must now change, such that N0P sleu is uiiectly followeu by uecouei


2u

mouule, since it must iun fiist in oiuei to ieassemble the payloau anu
tiansfei contiol to it.
The shellcoue now has the following conceptual stiuctuie:
|N0P SLEBj|BEC0BERj|ENCRYPTEB PAYL0ABj|RET0RN ABBRESSj
|BEC0BERj Since uecouei is now pait of the shellcoue, it must itself be
maskeu to avoiu being uetecteu by signatuiebaseu NIBS. The following
techniques accomplish uecouei polymoiphism: multiple coue paths
(peimuting machine instiuctions to peifoim the same opeiation in uiffeient
ways), ianuomly pauuing coue with junk N0P instiuctions, outofoiuei
uecouing anu ianuomly geneiateu instiuctions |2u,Suj.
In auuition to techniques alieauy mentioneu, it also seems ieasonable to
apply pauuing, ianuomize the oiuei of inuiviuual components as they aie
positioneu in memoiy by intiouucing auuitional jmp instiuctions in between
oi even use multiple uecoueis. The following aie some possible shellcoue
layouts |2uj:
|N0P SLEBj|ENCRYPTEB PAYL0ABj|BEC0BERj|RET0RN ABBRESSj
|N0P SLEBj|BEC0BER1j|ENCRYPTEB PAYL0ABj|BEC0BER2j|RET0RN ABBRESSj
|N0P SLEBj|ENCRYPTEB PAYL0ABj|PABBINuj|BEC0BERj|RET0RN ABBRESSj
The ability of shellcoue to tiansfoim into opeiationally equivalent vaiiants unueimines
the effectiveness of signatuiebaseu engines to iuentify a laige set of seiious netwoik
secuiity intiusions. uiven the natuie of polymoiphic coue, theie is veiy little piospect
in geneiating a single compiehensive iepiesentation of a given class of selfmouifying
exploit. Empiiical eviuence in one stuuy |2uj suggests that it woulu iequiie 2
240



21

signatuies to iepiesent a class of uecoueis alone (assuming that uecouei is Su bytes
long). Foi the sake of compaiison, the univeise only has 2
80
atoms. This means that foi
all piactical puiposes, hackeis have a limitless supply of uecoueis at theii uisposal. The
intiactability of moueling polymoiphic behavioi using stiingbaseu signatuie schemes
suggests that cieating a mouel of noimal behavioi patteins is a moie sensible puisuit. It
woulu seem that anomaly uetection NIBS woulu be bettei suiteu foi the task of
uetecting polymoiphic exploits. The uemanu foi signatuiebaseu uetection is expecteu
to ueciease as the utilization of selfmouifying shellcoue becomes moie pievalent.



22

Chapter2
SignatureDetection

Signatuiebaseu uetection, also known as misuse uetection, is knowleugebaseu; it
examines netwoik tiaffic foi patteins of wellknown attacks. It usually employs an
extensive upuateable catalog of attack signatuies as its knowleuge base to iecognize
attacks. 0nce new attacks aie uiscoveieu the unique pattein associateu with the attack
is encoueu anu auueu to the knowleuge base. An iueal signatuie is naiiow enough to
iepiesent the uefining chaiacteiistics of the attack scenaiio anu flexible enough to
encapsulate all possible vaiiations of the attack while not mislabeling goou uata.
0nce the uatabase is augmenteu with the new signatuie, the system becomes
aimeu with knowleuge iequiieu to uetect futuie instances of exactly that attack. The
pioblem aiises when the oiiginal attack foi which a signatuie exists has been slightly
mouifieu. Bepenuing on the specificity of the oiiginal signatuie oi the expiessiveness of
the unueilying mouel, the new vaiiation of the attack may no longei be uetectable by
the olu signatuie. Polymoiphic attacks, piesenteu in chaptei 1 piesent a seiious thieat
to signatuieuetection systems since they aie uesigneu to evaue specific signatuies.
0nfoitunately eveiy signatuiebaseu uetection system is only as goou as its
knowleuge base. In auuition the inability to uetect novel attacks foi which no signatuies
match is a concein. Time anu effoit aie iequiieu to constiuct anu iegulaily upuate the
signatuiebaseu pattein uatabase, so that by the time a signatuie is constiucteu, the
attack might be well unuei way. Fuitheimoie, as the uatabase giows laigei in size the


2S

oveiall system efficiency can be affecteu. All these conceins nonwithstanuing,
signatuie baseu systems aie a populai ueployment stiategy uue to the high accuiacy of
such systems.

2.1 Approaches
Common appioaches to implementeu signatuiebaseu intiusion systems will be biiefly
outlineu in this section. They incluue stiing matching, state moueling, expeit systems,
coloieu Petii nets, anu iulebaseu systems.

2.1.1 StringMatching
String matching is the most stiaightfoiwaiu type of signatuie uetection |2j. Each
intiusion scenaiio is iepiesenteu in some foim as an ASCII stiing oi some binaiy
pattein. The system scans some uata (e.g. incoming tiaffic (usually packet payloaus))
anu looks foi specific patteins. If a pattein is matcheu, the intiusion is consiueieu to
have taken place anu some action is tiiggeieu.

2.1.2 StateModeling
State moueling systems encoue each intiusion scenaiio as an oiueieu list of states anu
tiansitions |2,11j. Each unique state has the following conceptual stiuctuie:
|attiibute
1
, value
1
j |attiibute
2
, value
2
j | attiibute
n
, value
n
j
Each attiibute oi featuie is a system paiametei of inteiest monitoieu ovei time
by the IBS. Actions (oi events) that altei the values of these attiibutes initiate the


24

tiansition of the system fiom its cuiient state to the next. Each intiusion scenaiio is
iepiesenteu as a state machine (see figuie S).
Each noue iepiesents a state of the system. Aiiows uefine tiansitions between states.
The noimal state is a state of the system befoie an intiusion takes place. Failuie state
iepiesents the completion of the intiusion. Each inteimeuiate eiioi state is also a pait
of the intiusion scenaiio uesciibeu by the state machine.

2.1.3 ExpertSystem
Expert system uses a set of if-then implication iules to uesciibe known intiusion
scenaiios |1S,2,4,11j. Accumulateu uata fiom auuit events is conveiteu into
iepiesentational foim anu useu as facts. Knowleuge base of such system consists of fact
base anu iule base. At the heait of the expeit system is infeience engine which uiaws
conclusions fiom facts using iules, usually using techniques like foiwaiu chaining. Each
iule has a foim |anteceuentj|consequentj wheie antecedent is a list of patteins that
neeu to be matcheu with facts anu consequent is a list of actions that will be executeu if
the matching is successful.
When some set of facts fiom a fact base matches all patteins in anteceuent, a
binuing is cieateu foi each vaiiablevalue paii wheie values aie taken fiom facts that
Errur
1
Normal Errur
n
Failure
Figure S: Statebaseu Intiusion scenaiio


2S

have been matcheu. Among all such iule instantiations, the best one is pickeu using a
piocess calleu conflict iesolution. The binuings aie then applieu to consequent of that
iule anu the iule fiies. The fiiing will execute vaiious actions such as geneiating
secuiity aleits, teiminating usei session, auuing new facts to the knowleuge base oi
iemoving existing facts. The piocess keeps going until no moie iules can fiie.

2.1.4 ColoredPetriNets
Each attack scenaiio is encoueu as an instantiation of Coloieu Petii Automaton (CPA)
|1Sj. CPAs aie a lot moie expiessive than iegulai state machines: they pioviue
conuitional matching anu paitial oiueiing of events. CPA consists of states, tiansitions,
uiiecteu aics that connect states with tiansitions, anu tokens. CPA can have multiple
stait states anu one unique final state. CPA is initializeu with one token placeu in each
stait state. The attack is consiueieu to have take place when all tokens ieach final state.
Tiansitions can optionally be assigneu boolean expiessions calleu guards.
uuaius peimit assignments to CPA vaiiables, evaluation of conuitional opeiatois anu
calling of aibitiaiy functions. Conceptually tokens aie coloieu anu each coloi
coiiesponus to a unique set of patteins. Each event is taggeu with associateu
paiameteis oi facts. Patteins aie unifieu against facts in oiuei to geneiate vaiiable
binuings. These binuings aie then passeu to guaius foi evaluation.
Each tiansition is alloweu to have multiple input states as well as multiple
output states. A tiansition is enableu when each of its input states contains a token.




26

Foui conuitions must be satisfieu in oiuei to fiie a tiansition:
1. It must be enableu.
2. Specifieu tiansition event must fiie.
S. vaiiable binuings must successfully unify.
4. uuaius must be evaluateu to true.
When tiansition fiies, all its tokens, along with the infoimation they contain, aie
meigeu into one anu the geneiateu token is placeu in the output state. If theie is moie
than one output state, the token is uuplicateu anu a copy is placeu in each output state.
Example CPA is illustiateu in figuie 4. Theie aie 7 states anu S tiansitions. Each
tiansition has an associateu event label. Black tiansitions aie enableu since theii input
states contain tokens. Tiansition of event c
5
will be enableu when theie is a token in
both states s
3
anu s
6
. Aiioweu boxes symbolize guaius. This CPA iepiesents paitial
oiueiing of events, such that event c
1
necessaiily pieceues event c
2
anu event c
3

necessaiily pieceues event c
4
. Event c
5
is always the last in the sequence. This CPA also
maintains a list of global vaiiables. uuaius can use these vaiiables uuiing evaluation
anu peiiouically assign new values to them.

s
1
s
2
s
3

s
7

s
4
s
5
s
6

(this[FILE] == FILE)
PID = this[PID]
c
1
c
2

c
3
c
4
c
5

Variables:
FILE = usr/bin
PID =
(this[N] < 10) &&
(this[PID == PID])
Figure 4: CPA (iepiesenting paitial oiueiing of events)


27

2.1.5 RuleBasedSystems
Rulebaseu systems lack the powei of expeit systems but achieve gains in simplicity of
moueling intiusions |2j. Each intiusion scenaiio is iepiesenteu as a single iule of the
foim (conJ
1
A conJ
2
A A conJ
n
) - octions. Intiusion is consiueieu to have taken
place when each conuition is evaluateu to true. When a iule fiies in iesponse to
intiusion, all actions aie executeu (logging of intiusion, aleiting secuiity staff,
implementing iecoveiy mechanism, etc).
Snort is a fiee, ciossplatfoim, lightweight, iulebaseu intiusion uetection
system |28j. It is consiueieu to be a costeffective alteinative to expensive commeicial
NIBS systems. Snort is implementeu as a packet sniffei that monitois netwoik
inteiface in ieal time anu foi each inteicepteu packet peifoims content pattein
matching. Each Snort iule is a list of packet tests anu action which gets executeu when
all tests aie satisfieu. Action will eithei wiite the packet into a log oi geneiate an aleit.
Snort aichitectuie consists of packet uecouei, uetection engine anu
loggingaleiting subsystem |28j. Packet uecouei takes packets fiom uiffeient netwoiks
inteifaces anu foiwaius them to the uetection engine, wheie intiusions aie uetecteu.
Betection engine matches each packet against all iules until one fiies. Snort iules
allow matching of packet payloaus anu inuiviuual heauei fielus of all populai netwoik
(IP, ICNP), tianspoit (TCP, 0BP) anu applicationlayei (BNS, FTP, SNTP, etc) piotocols.
In the absence of any intiusion, the packet is uioppeu. The loggingaleiting subsystem
is iesponsible foi wiiting log files anu geneiating event notifications using usei
specifieu methous.


28

Snoit is a simple, yet poweiful intiusion uetection system capable of uetecting a
veiity of intiusions, incluuing stealth poit scans, buffei oveiflows, CuI attacks, anu
moie. It is easily ueployable on almost any noue on the netwoik anu iequiies minimal
auministiative maintenance |28j.



29

Chapter3
AnomalyDetection

Anomaly uetection systems (ABS) aie behaviorbased: they builu a iefeience mouel of
noimal system behavioi anu intiusions aie iuentifieu by uetecting ueviations fiom that
mouel. Anomaly uetection systems opeiate unuei a single assumption that if something
is abnoimal, it is suspicious. Foi example, an intiuuei anu legitimate usei of a system
aie likely to exhibit statistically uistinct behavioial patteins.
Anomaly uetection has two phases: tiaining phase anu uetection phase. In the
trainingphase the behavioi of the system is obseiveu in the absence of any intiusions,
anu piofile of noimal behavioi is cieateu. 0nlike signatuiebaseu uetection that
analyses inuiviuual netwoik events, anomaly uetection is also inteiesteu in leaining the
uynamic statistical piopeities of netwoik tiaffic, both on the global scale anu foi
inuiviuual netwoik noues. A piofile specifies how a netwoik is supposeu to behave in
the absence of any attacks. It incluues all leaineu statistical piopeities (uynamic
knowleuge) anu piotocol specifications (static knowleuge) |6j. In detection phase, a
piofile is compaieu against cuiient behavioi of the system anu any substantial
ueviations, oi anomalies, aie tieateu as inuicatois of a potential attack on the system.
Anomaly uetection offeis seveial auvantages ovei signatuie baseu intiusion
uetection systems.
Anomaly uetection systems have the ability to uetect unknown anu zeio
uay attacks. Intiusive activity tiiggeis an alaim not because the system


Su

iecognizes specific attack signatuie, but because the ueviation fiom noimal
activity is uetecteu.
Anomaly uetection offeis flexibility. 0nlike signatuiebaseu uetection
systems that usually iequiie a sepaiate signatuie foi eveiy unique attack
instantiation, a single anomaly can iepiesent an entiie class of attacks.
Not eveiy anomaly is tiiggeieu by malicious activities; some aie causeu by
malfunctioning netwoik uevices which can also be uetecteu by anomaly
uetection systems. Some examples of netwoik bieakuowns incluue netwoik
oveiloaus, file seivei failuies, congestions, bioaucast stoims, etc. Broadcast
storms aie conuitions in which a packet is bioaucasteu to all hosts on a
netwoik, anu each packet piompts a ieceiving host to iesponu by
bioaucasting its own packet that in tuin piompt fuithei iesponses, anu so
on. This snowball effect can have a seiious negative impact on netwoik
peifoimance. Network congestions usually occui uue to link oi noue failuies
which consequently iesult in all packets being ieiouteu on a uiffeient link,
causing excessive tiaffic loau anu substantially ieuucing system thioughput.
Anomaly uetection can also sometime suffei fiom seiious uiawbacks:
Befoie anomaly uetection system can be useu to uetect intiusive activity, it
must be tiaineu. Cieating a piofile of noimal netwoik behavioi is a
challenging task. Such piofile must be abstiacteu to incluue only those
featuies that aie necessaiy to uetect the types of attacks that the netwoik is
vulneiable to. Such infoimation is not always available in auvance anu is
iaiely complete. Extiaction of ielevant infoimative fiom tiaining uata tuins


S1

out to be a uifficult task because the typical behavioi of any computei
netwoik is chaotic anu coiielations between ielateu events aie often uifficult
to uiscein. Beficient piofile leaus to pooi peifoimance.
Piofile must be constantly upuateu since netwoik behavioi is uynamic; new
seivices aie auueu, new systems aie constantly intiouuceu, useis change
theii habits, auapt theii behavioi anu aie assigneu new tasks. ABS system
neeus to account foi this inheient instability by auaptively iecalibiating its
noimal uata mouel to ieflect the new enviionmental conuitions. Naintenance
of piofiles can be extiemely timeconsuming.
Anomaly uetection systems have pooi accuiacy of uetection chaiacteiizeu by
high iate of false positives (tenuency to geneiate an alaim in iesponse to
activity which is legitimate, yet abnoimal) anu false negatives (failuie to
uetect an intiusive activity which ueviates only slightly fiom noimal
behavioi). valuable time is often wasteu, iesponuing to false alaims.
In many piactical settings it is impossible to guaiantee that the tiaineu uata
is completely attackfiee. Patient attackeis often puiposefully tiain anomaly
uetection to giauually accept malicious behavioi as noimal.
When anomaly uetection system alaims netwoik staff about a possible
intiusion, it is often uifficult to ueteimine which specific event tiiggeieu the
alaim. This unceitainty iesults in uelayeu iesponse.



S2

3.1 FirstIntrusionDetectionModel
The fiist iealtime anomalybaseu intiusion uetection mouel was pioposeu by Boiothy
Benning in 1986 |Sj. The mouel was baseu on the hypothesis that secuiity intiusions
affecteu the state of the system by intiouucing anomalous activity patteins. Theiefoie
one coulu uetect intiusions by obseiving anu analyzing anomalies. The motivating
factois foi ueveloping iealtime intiusion uetection mouel weie |Sj:
1. Almost eveiy system suffeis fiom existing secuiity flaws which can be exploiteu
foi malicious puiposes, such as gaining access to sensitive uata, unueimining the
noimal functionality of seivices, monopolizing iesouices, etc.
2. Systems with known flaws cannot be easily ieplaceu without saciificing some
subset of theii functionality oi foi economic ieasons.
S. Beveloping new systems which aie 1uu% secuie is extiemely uifficult.
4. Even most secuie systems aie still vulneiable to piivilege abuses by authoiizeu
useis.
The pioposeu mouel is maue up of the following abstiact components:
Subjects: initiatois of actions (useis, piocesses, systems).
Objects: system iesouices (files, netwoik uevices, piogiams, uatabase
iecoius).
Audit records: actions peifoimeu by subjects on objects (file I0, login
attempts, uatabase iecoiu ietiieval).
Profiles mouel of noimal system activity, involving the behavioi of a set of
subjects with iespect to a set of objects. Piofiles contain a set of metiics anu
coiiesponuing statistical mouels. Netiics aie ianuom vaiiables iepiesenting


SS

quantitative measuies sampleu ovei a peiiou of time. Each metiic is a set of
sample points that iepiesent the value of a ianuom vaiiable at a paiticulai
time. These sample points aie feu into the statistical mouel in oiuei to
ueteimine if a new sample point iaises an anomaly conuition. Netiics coulu
be event counteis, inteival timeis between ielateu events, oi quantities
specifying consumption of a paiticulai iesouice.
uiven a ianuom vaiiable x sampleu n times to cieate sample points
x
1
, x
2
, , x
n
, statistical mouel ueteimines whethei a new sample point x
n+1
is
anomalous with iespect to x
1
, x
2
, , x
n
. Theie aie five pioposeu statistical
appioaches that can be auopteu by intiusion uetection system in oiuei
satisfy this goal:
1. Operational model a fixeu thiesholu that specifies a iange of values
consiueieu noimal is geneiateu manually using past expeiience
(sample points x
1
, x
2
, , x
n
aie often consiueieu). x
n+1
is ueemeu
abnoimal if its value falls outsiue noimal iange (numbei of successive
login failuies exceeus a ceitain thiesholu).
2. Confidence interval model mean p anu stanuaiu ueviation o of
sample points x
1
, x
2
, , x
n
is calculateu in oiuei to obtain a confiuence
inteival p _ J o. x
n+1
is ueemeu abnoimal if its value falls outsiue
this inteival. This mouel is moie flexible than opeiational mouel
because its abnoimality inteival is subjectuepenuent.
S. Multivariate model similai to confiuence inteival mouel except it
coiielates two oi moie ielateu metiics.


S4

4. Markov process model iepiesents each uistinct type of event as a
state anu uses state tiansition matiix to chaiacteiize fiequencies of
tiansitions between any two states. An event is consiueieu anomalous
if the piobability uefineu by pievious state anu tiansition ielation is
too low. Naikov piocess mouel is useu to uetect iiiegulaiities in event
sequences.
S. Time series model combines inteival timei anu event countei. Foi a
set of sample points x
1
, x
2
, , x
n
the mouel takes into account theii
values, oiuei anu aiiival times. x
n+1
is flaggeu as abnoimal if its
piobability of occuiiing at the time it is measuieu is too low. Time
seiies mouel is useful foi uetecting giauual but substantial vaiiations
in behavioi measuieu ovei time. Bowevei, it is moie computationally
expensive than pievious mouels.
Anomaly records iecoius cieateu in iesponse to an anomaly iaiseu with
iespect to a given piofile.
Activityrules conuitional iules specifying actions that neeu to be peifoimeu
when an auuit iecoiu oi anomaly iecoiu is geneiateu oi when a time peiiou
enus.

3.2 SourcesofAuditData
Theie aie seveial appioaches useu to peifoim anomaly uetection anu the choice of
appioach uepenus on the natuie of netwoik uata available foi analysis |27j. Theie aie


SS

vaiious souices that pioviue netwoik peifoimance infoimation that can be useu to
uetect anomalous netwoik events |6,27j:
Netwoik piobing tools such as ping oi traceroute collect netwoik
peifoimance measuies such as enutoenu uelays anu packet loss levels. Such
tools pioviue accuiate uata on the cuiient state of the netwoik.
Routing infoimation pioviues netwoik topology anu link utilization levels.
NIB vaiiables obtaineu fiom Simple Netwoik Nanagement Piotocol (SNNP)
aie countei vaiiables that measuie tiaffic infoimation at the inuiviuual
netwoik uevice. Pioviueu infoimation vaiies uepenuing on wheie the uevice
is locateu on the piotocol hieiaichy.
Netwoik anu tianspoit layei packet heaueis anu payloaus.
Netwoik tiaffic (tcpuump, NetFlow)
Tiaffic piobes that captuie anu analyze netwoik packets. Ntop is a 0NIX
baseu tiaffic piobe that pioviues a set of countei vaiiables that monitoi
vaiious netwoik activities, such as total tiaffic foi specific piotocols (volume
anu numbei of tiansmitteu packets), TCP session histoiy (uuiation,
tiansmitteu uata, iatio of fiagmenteu packets), iunning TCP0BP seivices
anu installeu opeiating systems, oveiall banuwiuth consumption, tiaffic
uistiibution (local vs. iemote), piotocol uistiibution (0BP vs. TCP), packet
uistiibution (in teims of size, IP vs. nonIP), etc |6j.



S6

3.3 ProtocolAnomalies
Protocolanomalies iefei to all exceptions ielateu to piotocol foimat anu behavioi with
iespect to typical piactical application. Netwoik piotocol is a set of iules goveining the
tiansmission of uata between computeis, applications, netwoiks anu inuiviuual
communication uevices. Nost piotocol specifications aie publisheu in RFCs anu similai
uocuments. Piotocols typically monitoieu incluue tianspoit layei piotocols (TCP, 0BP),
netwoik layei piotocols (IP, ICNP), anu application layei piotocols (BTTP, FTP). It is
impoitant to note that piotocols aie iaiely implementeu in piactice accoiuing to theii
official specifications. A mouel of noimal usage neeus to account foi this fact by
supeiposing official anu piactical stanuaius of usage. Inuiviuual packet heaueis aie
examineu to ueteimine if they obey official oi piactical guiuelines.
The oiuei in which the packets aie ieceiveu also matteis. Not all packets caiiy
actual uata; some packets aie iesponsible foi establishing new connections (SYN),
teaiing uown existing connections (FIN, RST), acknowleuging the ieceipt of uata (ACK),
etc. In auuition to specifying legal foimat foi each packet, piotocols uefine conuitions
unuei which it is peimissible to senu packets of vaiious types. Foi example, consiuei a
typical TCP session. Any TCP connection is establisheu by a pioceuuie calleu a three
wayhandshake, wheie a client senus a special contiol packet, calleu SYN packet, to the
seivei application iunning on a known poit, in iesponse to which the seivei senus
SYNACK packet which acts as an acknowleugment of connection iequest. Finally the
client senus its own acknowleugment back to the seivei. Following connection setup
phase is uata tiansfei phase uuiing which client anu seivei exchange application uata.


S7

TCP session is teiminateu by a fourwayhandshake uuiing which FIN packets anu theii
iespective acknowleugments aie tiansmitteu.
The following aie examples of piotocol anomalies |6,18j:
IP packets with spoofeu souice auuiess
IP packets wheie the souice anu uestination auuiess aie set to auuiess the
same uevice (LANB attack)
0utofsequence TCP packets
0nusually laige packets (ICNP Ping of Beath)
TCP packet that has unexpecteu oi piohibiteu combination of flags (packet
wheie SYN flag is set but belonging to a session alieauy in piogiess)
Fiagmenteu IP packets when fiagmentation is not iequiieu
Invaliu, oveilapping oi missing IP fiagments
Illegal packet flows (incomplete TCP thieeway hanushake that cieates a
halfopen connection. Coulu be a sign of a SYN scanning oi SYN floouing)
valiu packet sent by the unexpecteu netwoik agent (ICNP ieuiiect packet
sent by a host that isnt a ioutei)
Senuing nonBTTP uata to poit 8u
Running a seivice on a nonstanuaiu poit (BTTP packets aiiiving on poit SS)
Coiiupt checksums



S8

3.4 StatisticalAnomalies
Statisticalanomalies aie uetecteu by obseiving an aggiegate behavioi of netwoik tiaffic
(both globally anu at inuiviuual netwoik uevices) ovei a peiiou of time. In the absence
of an attack, theie is a stable balance among uiffeient types of outgoing anu incoming
packets: TCP FIN, TCP SYN, TCP uata, ICNP echo iequestieply, etc. Ceitain attacks,
such as BoS attacks, will affect tiaffic patteins iecoiueu uuiing tiaining phase anu will
iesult in statistical anomalies. Tiaffic patteins aie typically monitoieu by netwoik
management piotocols anu tiaffic piobes.
Some examples of statistical anomalies incluue |6,18,27j:
0nusually high volume of 0BP tiaffic ielative to TCP tiaffic
Buist in the fiequency of timeouts (connections expiiing uue to inactivity)
0nusually high volume of SYN packets ielative to othei types of tiaffic (SYN
floou)
Excessive tiaffic to the mail seivei (Possible BoS attack)
Rise in the numbei of connection attempts maue to a paiticulai poit
0nusually high ICNP echoieply iatio (sign of netwoik piobing)

3.5 Approaches
At this point we piesent seveial case stuuies that uelineate vaiious anomaly uetection
appioaches useu in piactice.



S9

3.5.1 ProtocolStateModeling
Nost known piotocols can be conceptually iepiesenteu as finite state machines. Noie
foimally, piotocols aie moueleu using extenueu finite state automata (EFSA) |18j,
which uiffei fiom tiauitional finite state automata in two iespects: (1) events of EFSA
may have aiguments, anu (2) it can have a finite set of state vaiiables. Each EFSA has a
list of contiol states {s
1
, , s
n
], stait state s, final state , anu a list of state
vaiiables {:
1
, , :
n
]. Tiansitions aie uiiecteu aics, connecting states.
TCP piotocol is the most wiuely useu connectionoiienteu tianspoit piotocol. A
sepaiate piotocol state machine is maintaineu foi each active TCP connection. States
iepiesent vaiious connection stages while tiansitions iepiesent the event of ieceiving a
paiticulai type of packet oi timeout. Foi example, when client anu seivei complete
theii thieeway hanushake, the TCP connection enteis the ESTABLISBEB state. If the
seivei ieceives a FIN packet iequesting connection to be teiminateu, its TCP piotocol
machine will entei FIN_WAIT_1 state. Foi a complete uesciiption of states anu
tiansitions of TCP piotocol, see figuie S.
A transition relation has a foim c(x
1
, , x
n
)|conJ - |octions, stotc] wheie c is
an event, vaiiables x
1
, , x
n
aie aiguments of that event, conJ is a boolean expiession
that involves state vaiiables, event aiguments anu cuiient contiol state. octions anu
stotc aie a list of actions that will be executeu anu a state the EFSA will entei,
iespectively, when event c occuis anu conJ evaluates to true. Possible actions incluue
assignments to state vaiiables anu invocations of exteinal functions.




4u



When monitoiing behavioi of a ceitain piotocol, multiple instances of state
machines aie cieateu, one foi each active connection. Connection is uefineu by the
souice IPpoit anu uestination IPpoit. When a packet aiiives, it is foiwaiueu to all
existing state machines. Bowevei, only the state machine whose connection matches
the souice of the packet will fiie the appiopiiate tiansitions. If a packet initiates a new
LISTEN
SYN_RCVD SYN_SENT
ESTABLISHED
FIN_WAIT_1
FIN_WAIT_2
CLOSING
TIMED_WAIT
CLOSE_WAIT
LAST_ACK
CLOSED
SYN,SYNACK
SYN
RST
FIN
ACK SYNACK,ACK
FIN FIN,ACK
FINACKACK FIN
FIN,ACK
ACK FIN
ACK
Timeout
FIN,ACK
CLOSED
SYN
SYN,SYNACK
Figure S: TCP State Nachine


41

connection iequest, a new state machine is allocateu to keep tiack of the new
connection. A tiace is uefineu as a sequence of states visiteu by a given EFSA uuiing its
lifetime along with coiiesponuing state vaiiable values.
The following statistics can be obtaineu fiom tiaces of IP state machine |18j:
Fiequency with which each tiansition is taken
Nost commonly encounteieu value of a paiticulai state vaiiable at a
paiticulai state
Bistiibution of values of state vaiiables
Tiansitions taken by a state machine of a given tiace iepiesent two types of impoitant
events |18j: (1) the ieception of unexpecteu packet anu (2) timeout event which means
that expecteu packet was not ieceiveu. Both events suggest a possible netwoik failuie
oi attack. Two piopeities ielateu to inuiviuual tiansitions aie iuentifieu |18j: (1)
whethei a given tiansition is taken by a tiace anu (2) the value of a given state vaiiable
oi a packet fielu when a tiansition is taken. This infoimation can be iepiesenteu as
aveiage values, but uue to chaotic natuie of netwoik behavioi, captuiing uistiibutions
is moie uesiiable as it pioviues a moie accuiate measuie of netwoik activity. Type (1)
piopeities aie captuieu as fiequency uistiibutions, wheieas type (2) piopeities aie
captuieu as uistiibutions of values of state vaiiables. If values aie categoiical (IP
auuiess), as oppose to scalai (packet fielu size), they aie iepiesenteu as uisciete
counteis. Bistiibutions aie measuieu on multiple timescales that iange fiom
milliseconus to thousanus of seconus. This pioviues a balance between fast uetection of
iapiuly piogiessive attacks anu uelayeu but moie accuiate uetection of slowei attacks.


42

Ceitain statistics aie specifically tailoieu to uetect wellknown attacks. Foi
example, the numbei of unique IP auuiesses foi which the packets weie ieceiveu in the
last t seconus, wheie t is a small time fiame, anu the fiequency of timeout tiansitions in
PKT_RCvB (packet ieceiveu) state of IP state machine, aie both useful statistics foi
uetecting ping sweeps |18j.
Piotocolbaseu anomaly uetection systems geneially have high uetection iates of
both known anu unknown attacks anu low false positive iates. 0ne of the main benefits
of such systems is simplifieu featuie selection. Nost attacks can be uetecteu by simply
obseiving the uistiibution of fiequencies with which each tiansition is taken in a state
machine.

3.5.2 SignalProcessingofMIBVariables
Simple Netwoik Nanagement Piotocol (SNNP) is implementeu as an application layei
piotocol that iuns ovei 0BP anu pioviues facilities foi exchanging management
infoimation between netwoiking uevices. SNNP allows netwoik auministiatois to
monitoi netwoik health, as well as uetect anu iesolve netwoik peifoimance issues. An
SNNPmanageu netwoik consists of thiee vital components: netwoik uevices that neeu
to be monitoieu, SNNP agents, anu SNNP managei. Nanageu uevices aie inuiviuual
netwoik noues such as iouteis, switches, hubs, etc. Agents aie softwaie applications
that iesiue in a manageu uevice anu collect netwoik management infoimation which
they then communicate to SNNP managei using SNNP piotocol.
Eveiy netwoik uevice stoies a set of NIB (Nanagement Infoimation Base)
vaiiables that aie specific to its functionality anu implementeu as counteis. Netwoik


4S

uevices aie classifieu in teims of how fai up the piotocol stack they opeiate. Foi
example, iouteis aie netwoiklayei uevices while biiuges aie linklayei uevices.
Statistical analysis of NIB vaiiables allows netwoik auministiatois to uetect
many types of netwoikpeifoimance anomalies in an effoit to anticipate anu effectively
pievent netwoikwiue failuies. In one stuuy |27j, signal piocessing methou was useu to
uetect netwoik anomalies by moueling coiielateu abiupt changes in time seiies
geneiateu fiom thiee NIB vaiiables chosen fiom IPlayei gioup: ipIR (the total numbei
of uatagiams ieceiveu fiom all the inteifaces of the ioutei), ipIBe (the numbei of
uatagiams foiwaiueu to the highei layeis) anu ip0R (the numbei of uatagiams
ieceiveu fiom the highei levels). Foui types of netwoik anomalies weie uetecteu using
this appioach: file seivei failuies, piotocol implementation eiiois, netwoik access
pioblems anu iunaway piocesses.

3.5.3 DataMiningUsingClusters
0nuei noimal opeiation, netwoik auaptei only piocesses packets auuiesseu to its
unique NAC auuiess. When an auaptei is switcheu to promiscuousmode, it inteicepts all
packets passing thiough its netwoik anu foiwaius them to the uppei levels of piotocol
stack. tcpdump is a poweiful packet sniffei that logs the heaueis of all packets that
aiiive at the netwoik inteiface. vast volume of tiaining uata can be collecteu this way
anu useu to geneiate a piofile of noimal netwoik behavioi. Bowevei, extiacting
ielevant infoimation fiom a laige amount of multiuimensional iaw uata is a uifficult
anu often computationally expensive task.


44

Bata mining iefeis to methous anu algoiithms useu to analyze uata in oiuei to
leain about its chaiacteiistic piopeities. In othei woius, uata mining is a piocess of
extiacting knowleuge fiom uata. In the context of netwoik anomaly uetection, uata
mining can seive two impoitant functions: tiaffic classification anu outliei uetection.
Traffic classification is a piocess of uefining patteins that aie typical foi a paiticulai
type of netwoik tiaffic, foi example noimal BTTP tiaffic, tiaffic obseiveu uuiing ICNP
floou oi tiaffic geneiateu by SYN piobing. Outlierdetection iefeis to iuentifying singulai
uata objects that uo not belong to any existing tiaffic piofile anu theiefoie tieateu as
anomalies. In this section we examine a uata mining methouology calleu clusteiing.
Clustering iefeis to methous anu algoiithms that paitition a set of uata points
into a finite set of clusteis. A cluster is an aggiegate of uata objects, which aie assumeu
to be similai to one anothei within the same clustei, anu uissimilai to uata objects of
othei clusteis. A goou clusteiing scheme will piouuce clusteis with high intiaclass
similaiity anu low inteiclass similaiity. Theie aie many classes of clusteiing
algoiithms uesciibeu in liteiatuie: paititioning algoiithms, hieiaichical algoiithms,
uensitybaseu algoiithms anu giiubaseu methous |14j. Foi example, densitybased
clustering algoiithms assume that clusteis aie iegions of high uensity in the uata space
suiiounueu by iegions of low uensity. Foi each uata point, the algoiithm computes the
uensity of its neighboihoou within a ceitain iauius. All neighboihoous of high uensity
(iegions wheie numbei of uata points exceeus some minimal thiesholu) aie then
aggiegateu togethei to foim clusteis |14j. 0ne common technique is to paitition the
uata space into a giiu of cells, compute the uensity of inuiviuual cells anu then meige


4S

cells togethei to foim clusteis. This appioach ieuuceu piocessing time allowing the
algoiithm to scale to laigei sets of uata.
In one stuuy |SSj, Kmeans paititional clusteiing algoiithm was useu to uetect
netwoik anomalies by classifying flow iecoius piouuceu by Cisco NetFlow piotocol. A
flow iefeis to a uniuiiectional stieam of IP packets iuentifieu by theii souice IPpoit,
uestination IPpoit anu tianspoit piotocol (TCP, 0BP). Each flow iecoiu also incluues
associateu statistical piopeities such as total numbei of packets anu bytes tiansmitteu
at specific time inteivals.
In oiuei to piouuce tiaining uata foi Kmeans clusteiing algoiithm, flow iecoius
neeu to be piepiocesseu anu tiansfoimeu. Fiist, flow iecoius aie gioupeu into seivice
classes accoiuing to utilizeu piotocol anu poit numbeis ieseiveu foi commonly useu
seivices. Foi example, web seivei iuns on poit 8u ovei TCP (WebBTTP seivice class)
while BNS uses 0BP on poit SS (BNS seivice class). The ieason foi this classification is
that chaiacteiistic piopeities of noimal tiaffic vaiy acioss seivices. Kmeans algoiithm
is thus applieu sepaiately foi each seivice class. Next, flow iecoius belonging to
inuiviuual classes aie aggiegateu anu tiansfoimeu into uatasets foi equally spaceu time
inteivals. The following featuies aie uefineu foi each uataset:
1. Total numbei of packets sent to oi fiom a given poit in the consiueieu time
inteival
2. Total numbei of bytes sent to oi fiom a given poit in the consiueieu time
inteival
S. Numbei of unique souiceuestination paiis obseiveu in the consiueieu time
inteival.


46

A
B
C
Normal
Anomalous
Figure 6: Anomaly uetection using clusteis
J
max

(1) anu (2) facilitate the uetection of tiafficvolume anomalies (Benial of Seivice) while
(S) helps uetect anomalies associateu with netwoik piobing (ICNP ping, SYN poit scan)
anu uistiibuteu attacks.
Kmeans is an iteiative clusteiing algoiithm that paititions uata objects into K
uisjoint clusteis within theii featuie space. The algoiithm follows these steps:
1. Befine K aibitiaiily chosen centioius (mean points).
2. ueneiate a set of clusteis by assigning all uata points to the neaiest centioiu.
S. Recalibiate the position of each centioiu by moving it to the centei of its
iespective clustei.
4. Repeat steps 2 anu S until centioius conveige, foiming final clusteis.
Since anomalous tiaffic anu noimal tiaffic aie assumeu to be chaiacteiistically
uiffeient, K is chosen to be 2, meaning Kmeans algoiithm outputs at most two
centioius, one foi each type of tiaffic.
0nce clusteis aie geneiateu anu
manually labeleu as eithei noimal oi
anomalous, netwoik anomalies can be
uetecteu using uistancebaseu
classification anu outliei uetection. A
uata object is classifieu as an anomaly if
it is closei to the anomalous centioiu
than to the noimal centioiu, oi if its
uistance to the noimal centioiu is laigei than the pieuefineu thiesholu J
max
, in which
case it is tieateu as an outliei. In figuie 6 point A is tieateu as anomaly because it is


47

outsiue J
max
anu point C is tieateu as anomaly because it is closei to anomalous
centioiu.

3.5.4 DetectingSYNFloods
Accoiuing to CSIFBI Cybeiciime Suivey Repoit of 2uuS, BoS weie iesponsible foi a
loss of $6S million. TCP SYN floou is the most pievalent type of BoS attack |19j.
Theiefoie it is ciucial to have effective means of uetecting anu uealing with SYN floous
as eaily as possible. Auaptive thiesholu algoiithm is a simple algoiithm that facilitates
the uetection of SYN floou attacks by testing whethei the numbei of SYN packets
piesent in netwoik tiaffic ovei a given time inteival exceeus a ceitain thiesholu. The
value of a thiesholu is auaptively mouifieu to account foi iecent behavioi uefineu by
exponentially weighteu moving aveiage. Auaptive thiesholu algoiithm peifoims veiy
well foi high intensity attacks, yieluing high accuiacy, low false alaim iate anu low
uetection uelay |19j.
Foi low intensity attacks howevei, the algoiithms peifoimance is significantly
ueteiioiateu uue to its limiteu ability to maintain past histoiy of thiesholu violations,
piouucing high false alaim iate. Nove auvanceu algoiithms exist that peifoim equally
well foi uetection of low anu high intensity attacks, such as C0S0N (cumulative sum)
algoiithm baseu on hypothesis testing |19j.



48

Signature
Sensor
Anomaly
Sensor
Network
traffic
Attack
detected
Anomaly
detected
Noattack
Normal
traffic
Figure 7: Bybiiu IBS
3.6 HybridIntrusionDetection
It is impoitant to note that a ieliable anu
iobust intiusion uetection system shoulu
combine both anomaly anu signatuiebaseu
uetection. System that enjoys the
complementaiy benefits of both types of
uetection methouologies will be chaiacteiizeu
by high accuiacy, ability to uetect novel
attacks anu ieuuceu iate of false alaims.
Intiusion uetection system that integiates
anomaly uetection anu signatuie uetection is
calleu hybriddetectionsystem |7j.
Such appioach uiviues uetection into
two stages. In the fiist stage, netwoik tiaffic
passes thiough signatuie sensoi. Alaim is geneiateu in the piesence of an attack
(assuming the attack is known in auvance). 0theiwise, the tiaffic is consiueieu clean at
least as fai as signatuie uetection is conceineu. In the seconu stage, the same tiaffic is
feu into anomaly sensoi. If the anomaly is uetecteu, the alaim is geneiateu. See figuie 7.
Bybiiu intiusion uetection is in piincipal a much moie effective solution to
netwoik suiveillance. Bowevei, uiffeient intiusion uetection technologies examine
netwoik tiaffic in veiy specific ways anu aie configuieu to opeiate in iestiicteu highly
tuneu netwoik setups. The majoi challenge unueilying the piactice of builuing an
opeiational hybiiu system is getting its component systems to effectively inteiopeiate.


49

Chapter4
Response

Betecting intiusions is only the fiist step towaius secuiing the netwoik. Attack
uetection by itself is of limiteu use if no measuies aie implementeu to issue a iesponse
oi initiate some foim of iecoveiy mechanism fiom malicious activity. Intiusion
Response Systems (IRS) obtain misuse oi anomaly iepoits fiom the IBS systems in
oiuei to ueciue how to effectively thwait the attack anu ensuie the safety of computing
assets. Bistoiically, intiusion iesponse has ieceiveu a lot less attention than intiusion
uetection because of the neeu to involve a live opeiatoi in the uecision loop. But as
attacks piolifeiate in sophistication, speeu, anu inciuence, it becomes moie anu moie
costly to neglect this aiea of netwoik infiastiuctuie.
Theie aie two geneial categoiies of iesponse: active anu passive |17j. Passive
responsesystems make no attempt to eliminate the attacks oi minimize uamages causeu
by them. Theii only job is to log the intiusion oi foiwaiu a note to the netwoik
management staff (e.g. via email, pagei, cell phones) who woulu then have to ueciue
how to iesponu. Intiusion iepoits might contain infoimation such as attack taiget iu,
time of attack, seveiity level, infoimation on specific packets useu to gain unauthoiizeu
access, IP auuiess of the attackei, anomaly statistics, etc |2Sj. Nost intiusion uetection
systems have histoiically implementeu a passive iesponse mechanism. This can be
attiibuteu to the high false positive iates of eaily IBS systems anu the bias towaius
incluuing a human opeiatoi in the uecision loop.


Su

Active response systems, on the othei hanu, attempt to countei an attack by
taking some foim of evasive oi coiiective action. Active iesponse systems aie fuithei
classifieu in teims of theii level of automation |2Sj. Manualresponsesystems geneiate a
list of possible counteimeasuies but the final uecision iests with the netwoik
management staff. The main auvantage of iequiiing human inteivention is that many
systems have been known to geneiate a high iate of false alaims. It woulu be countei
piouuctive anu often uetiimental to the effective opeiation of the netwoik if system
took potentially aggiessive action in iesponse to eveiy uetecteu anomaly, such as
closing poits oi teiminating live connections. Shoulu attackeis themselves leain this,
they will gain the ability to tiiggei a uenial of seivice to such systems by meiely
attempting an intiusion. Bowevei, ceitain netwoiks suffei fiom high inciuence of
malicious activity anu iequiiing human inteivention on eveiy attack might be too
costly. Netwoik auministiatois aie known to be iathei busy anu uelayeu iesponses
pioviue attackeis with a sufficient winuow of oppoitunity to succeeu in theii attack
|17j. Stuuies show that if the time gap between uetection anu iesponse is 1u houis,
skilleu human attackeis have an 8u% success piobability |S6j. 0n the othei hanu,
automateu exploits often iequiie meie minutes oi even seconus to biing about
iiiepaiable uamage |S7j. In auuition the type of iesouice being piotecteu is a factoi in
the uecision. Piotecting an ecommeice website is uiffeient than piotecting a nucleai
weapons netwoik.
Autonomous response systems automatically ueciue on anu execute an
appiopiiate iesponse policy. Examples of common iesponse actions aie: mouifying
fiiewall iules anu access contiol lists (ACL), blocking poits anu IP auuiesses,


S1

teiminating TCP connections, tiacing the souice of the attack, iestaiting the taiget,
uiveiging suspicious TCP connection fiom the taiget system to a tiap system |2S,26j,
etc. See section 7.1 foi auuitional examples.
Static mapping response systems use uecision tables to uiiectly map an attack
scenaiio to haiucoueu iesponse. Such appioach is contextinuepenuent as it fails to
take into consiueiation the unique ciicumstances unuei which the attack was tiiggeieu.
Foi example, taking the system offline is a possible iesponse to any intiusion, but if a
system in question is the mail seivei of a laige oiganization, taking it offline may piove
to have uisastious consequences. If on the othei hanu, a backup seivei exists which can
peifoim the same uuties, then uisconnecting the piimaiy seivei in case it was
compiomiseu may not tuin out to be such a bau iuea.
Dynamic mapping response systems iesponu to an attack by consiueiing
ciicumstantial eviuence suiiounuing the attack. Such systems aie implementeu as iule
baseu uecision engines anu expeit systems. Each iule is a nesteu hieiaichy of
implications that ueciue actions baseu on the following factois |17,2S,S7j:
Type of attack (piobing, uenial of seivice)
Confiuence level (how many monitoieu featuies substantiate that the attack
is taking place.)
Inciuent seveiity
Reliability of intiusion uetectoi that tiiggeieu the alaim (may uepenu on
false positive iate)
The utility of the taiget system foi useis anu the oiganization


S2

Type of peipetiatoi (sciipt kiuuie vs. piofessional ciackei, exteinal paity oi
insiuei)
Piivileges associateu with the compiomiseu usei account (if applicable)
Expeit peipetiatois aie peisistent anu iesouiceful; if theii attack fails initially, they will
auapt theii tactic so as to avoiu the uefense anu tiy again. So the iesponse system itself
must auapt its stiategy.



SS

Chapter5
NIDSforDistributedNetworks

Nost commeicial intiusion uetection aichitectuies suffei fiom theii lack of ability to
scale well to laige netwoiking enviionments. They tenu to have centializeu auuit
collection anu stoiage mechanisms that place heavy buiuen on available computational
iesouices. As netwoiks giow in size by incieasing the numbei of inuiviuual uevices anu
auuing new seivices, it becomes incieasingly uifficult to manage the eveigiowing
knowleuge iepositoiies. The statistical analysis engines as well as signatuie uetection
engines have to soit thiough massive amounts of auuit uata in oiuei to iuentify
anomalous oi malicious behavioi. As a iesult peifoimance substantially ueclines,
attacks aie faileu to be uetecteu in ieal time uue to oveiheau in uata piocessing, the
instance of false alaims iises uue to excessive noise. In auuition, the centialization of
intiusion uetection pioviues a single point of failuie. This chaptei intiouuces two
netwoikbaseu commeicial intiusion uetection systems that can scale to laige
enteipiise netwoiks.

5.1 EMERALD
The ENERALB pioject intiouuces a uistiibuteu, scalable, extensible, inteiopeiable
intiusion uetection system uesigneu foi laige enteipiise netwoiks |16j. ENERALB
pioviues a complementaiy analysis ovei the opeiation of netwoiks anu inuiviuual


S4

entities within them by combining signatuiebaseu uetection anu statistical piofiling to
uetect anomalies. At the heait of ENERALB IBS is a layeieu hieiaichy that uiviues the
task of netwoik suiveillance among thiee uiffeient types of monitois. Seivice monitois
aie iesponsible foi inuiviuual netwoik components anu seivices within a single
uomain. Bomain monitois hanule intiusion uetection in the context of entiie uomain.
Enteipiise monitois pioviue global piotection acioss all monitoieu uomains anu
facilitate the uetection of netwoikwiue thieats such as inteinet woims anu laige
cooiuinateu attacks. Ceitain types of intiusions manifest globally anu aie not maue
visible by monitoiing inuiviuual netwoik assets.
Bata collecteu on the lowei levels of the suiveillance hieiaichy is tiansmitteu to
uppei levels by a subsciiption mechanism. Seivice monitois collect auuit uata fiom
inuiviuual components anu compile analysis iesults which then get piopagateu up to
uomain monitoi, which in tuin peifoims coiielateu analysis of uata ieceiveu fiom all its
seivice monitois. Analysis iepoits fiom multiple uomain monitois aie then sent to an
enteipiiselevel monitoi. Such hieiaichical appioach to intiusion uetection uoes not
impose heavy buiuen on computational iesouices because the iesponsibility of uata
collection anu analysis is uiviueu among multiple monitois. This facilitates timely
uetection anu consequently, eaily iesponse to thieats. Fuitheimoie, since each monitoi
is iesponsible foi a uata stieam belonging to each inuiviuual netwoik component, it is
easiei to filtei out noise anu peifoim a moie focuseu anu finegiaineu uata analysis. Foi
example, each monitoi neeus to maintain only those signatuies anu statistical piofiles
which aie ielevant to the taiget being monitoieu. A taiget coulu be anything: a host, a
seivice, a ioutei, a uomain, an entiie netwoik, etc.


SS

5.1.1 EMERALDMonitorArchitecture
Each monitoi, iegaiuless of its scope (seivice, uomain oi enteipiise) has foui essential
components: piofilei engine, signatuie engine, iesolvei anu iesouice object |16j.
Piofilei engine peifoims statistical analysis on the event stieam in oiuei to
classify anomalous activity. Nouulaiity is achieveu by logically sepaiating piofile
management fiom statistical fiamewoik foi anomaly scoiing. ENERALB Signatuie
engine employs a iulebaseu infeience signatuie uetection scheme. Each inuiviuual
knowleuge base is specifically tailoieu foi a single analysis taiget. This ieuuces noise
iatio, incieases accuiacy of uetection anu impioves the oveiall peifoimance.
Resolvei is ENERALBs counteimeasuie expeit system. It piocesses intiusion
anu anomaly iepoits, implements an appiopiiate iesponse policy, manages inteinal
analysis engines anu inteifaces with exteinal engines thiough a subsciiption seivice in
oiuei to coiielate analysis iesults anu uisseminate intiusion iepoits to othei monitois.
Response policy is ueteimineu by infeiiing the confiuence level of the attack anu the
level of its seveiity. Resolvei coulu iesponu by geneiating an intiusion aleit, closing a
connection, teiminating a piocess, etc.
Resouice object is a pluggable libiaiy that contains taigetspecific uata
stiuctuies, functions anu configuiation vaiiables. This component is what uiffeientiates
one monitoi fiom the next. Both the iesolvei anu analysis engines aie inuepenuent of
the taiget on which the monitoi is ueployeu. Resouice object specifies uata foimat,
analysis iesults syntax anu semantics, statistical thiesholus, intiusion signatuies,
filteiing ioutines that conveit taigetunique events into geneial foim, subsciiption lists,
iesponse hanuleis, etc.


S6

5.2 GrIDS
uiIBS is a iealtime giaphbaseu netwoik intiusion uetection system |29j. It is uesigneu
to scale well to laige netwoik enviionments anu pioviue uetection capabilities against
laigescale cooiuinateu oi automateu attacks, especially inteinet woims. uiIBS views a
netwoik as a collection of useis, hosts anu uepaitments that communicate via paiiwise
netwoik connections uefineu by theii application piotocol (BTTP, TELNET, etc). Each
uepaitment itself is a collection of otheis uepaitments, useis anu hosts. Attack
scenaiios aie iepiesenteu as activitygraphs.
Activity giaphs uefine tiaffic patteins among inuiviuual netwoik entities ovei a
given peiiou of time. Each giaph consists of noues anu uiiecteu euges. Noues iepiesent
hosts oi uepaitments, while euges iepiesent netwoik tiaffic between them. Foi
example when host A initiates connection to host B, an euge leauing fiom A to B is
geneiateu. Biffeient attacks geneiate chaiacteiistic types of giaphs. Foi example,
woims aie malicious piogiams that piopagate acioss the netwoik by invauing one
machine anu then using its iesouices to invaue its neighbois |29j. Tiaffic piouuceu by
woim piopagation foims a tieelike activity giaph. Example giaph is illustiateu in
figuie 8.



S7


Noues, euges anu giaphs aie uefineu with supplementaiy attiibutes. Ceitain
attiibutes of the giaph, in paiticulai size, uepth anu bianching factoi, can be useu to
make infeiences about the confiuence oi seveiity of the attack. uiIBS, like ENERALB,
featuies a multilayei suiveillance hieiaichy. Each uepaitment aggiegates hosts as well
as othei uepaitments anu has its own giaphbuiluing engine, which evaluates activity
giaphs within the uepaitment. In oiuei to mouel activity among uepaitments on the
same level of hieiaichy, those uepaitments aie ieuuceu into a single noue. Figuie 9
uemonstiateu how uepaitments aie vieweu fiom a highei level. The topology of
ieuuceu uepaitments is lost at least as fai as highei level mouule is conceineu, but theii
attiibutes (size, uepth anu bianching factoi) which uo get passeu up, aie much moie
ielevant foi the puiposes of iuentifying intiusions.
D
F
G
J
K
I
L
M
B
E
H
A
C
Figure 8: Woim activity giaph


S8


uiIBS uata souices aie mouules that monitoi activity on inuiviuual hosts anu
netwoiks anu senu activity iepoits, in a foim of a uiscoveieu noue oi new euge, to the
giaph engine which then incoipoiates the new infoimation into its active giaphs.
Nouules come in a foim of packet sniffeis oi exteinal NIBS systems spieau aiounu the
netwoik. When the engine on the lowei level builus a giaph, it geneiates a summaiy
anu passes it to the engine foi its paients uepaitment, which will in tuin incoipoiate
new infoimation into its own giaph.
uiaphs aie geneiateu in accoiuance to useispecifieu iule sets. A iule set is a
foimal specification of one kinu of giaph. Rules specify conuitions unuei which an
activity iepoit will be incoipoiateu into a giaph stiuctuie, thiesholus that ueteimine
when the giaph shoulu be tieateu as a sign of intiusive activity anu which actions to
take in iesponse, as well as conuitions unuei which oveilapping giaphs can be
combineu.

A
B
C E
D
E
G
H
I
J
AB
N=2
CG
N=5
HJ
N=3
Figure 9: Reuucing uepaitments into a single noue


S9

Chapter6
EfficiencyEvaluation

In oiuei to be an effective anu ieliable secuiity solution, intiusion uetection system
must fulfill ceitain ciiteiia. In oiuei to assess how intiusion uetection system peifoims
on a given netwoik, the following metiics aie often consiueieu |4j:
Accuracy of detection: iueally intiusion uetection system woulu flag all
illegitimate behavioi as intiusive anu not alaim auministiatois when
uetecting behavioi which is chaiacteiistically uiffeient fiom noimal, but
nonetheless legitimate. False positive occuis when IBS iepoits an intiusion
when theie is none. False negative occuis when an intiusion fails to be
iepoiteu.
Completeness: it is uesiiable, though iaiely achievable in piactice, to uesign
NIBS that successfully uetects all attacks piesent in netwoik tiaffic. Since
peifect completeness cannot be ieacheu, we quantify how closely NIBS
appioaches this iueal maik.
Performance measuies how effectively IBS piocesses auuit uata. Pooi
peifoimance iesults in uelayeu attack uetection. It is impoitant to uetect all
intiusions in iealtime befoie any seiious uamage is uone to the netwoik.
Belayeu uetection iesults often iesults in significant loses.
Fault tolerance iefeis to the ability of IBS to iesist attacks against itself. A
netwoik wheie IBS itself is vulneiable to attacks is in many instances as


6u

insecuie as a netwoik with no IBS at all. Foi example, if a skilleu attackei
locates unpiotecteu knowleuge iepositoiy that contains attack signatuies, he
coulu uelete signatuies that uetect specific attacks anu then caiiy out those
attacks, avoiuing any uetection.
0ne class of attacks against IBS is calleu ciash attacks |Sj. Such
attacks attempt to unueimine the opeiation of IBS by causing it to ciash oi
entei a piocessing jam. Fault toleiant system has mechanisms in place that
minimize the iisk of being uisableu by the attackei. Foi example, Bio is a
iealtime netwoik intiusion uetection system that implements a fault
iesistant mechanism known as the watchuog timei. This timei is activateu
when Bio begins piocessing a new event. If the system is still piocessing the
same event when the timei expiies, it assumes to be in a piocessing loop so it
uiops cuiient event anu pioceeus to analyze the next one.
Ceitain Benial of Seivice attacks specifically taiget intiusion uetection
systems. This is accomplisheu by geneiating a laige quantity of auuit uata
anu feeuing it uiiectly into the event monitoi of the IBS. If enough uata is
geneiateu this way, the analysis engine will be oveiwhelmeu anu iealtime
uetection will be effectively uisableu. 0ne possible solution |Sj is to alleviate
the loau by uiscaiuing uata flows foi which the state is maintaineu, yet no
ieal piogiess is maue. In essence, the system is making a uecision to ignoie
uata which is assumeu to have low piioiity in oiuei to effectively piocess
new uata.



61

6.1 Benchmarking
The behavioi of each netwoik is a function of its opeiational uomain (ieseaich
enteipiise, euucational enteipiise, commeicial enteipiise, business enteipiise, etc), its
useis (theii ioles, expeiience level, habits, etc), applications (ecommeice, gaming,
www, stieaming viueo, etc). Each netwoik enviionment has uistinct chaiacteiistics that
might affect the peifoimance of anomaly uetection systems ueployeu on it. These
chaiacteiistics incluue tiaffic piofiles (0BP vs. TCP), tiaffic volume, numbei of
connection iequests pei seconu, ielative volume of incoming anu outgoing tiaffic, most
fiequently useu applications (BTTP, FTP, Nail), etc.

6.1.1 CaseStudy:BenchmarkingBasedonDataEntropy
Foi anomaly uetection systems that implement statistical moueling, one chaiacteiistic
of paiticulai impoitance is uata entiopy oi iiiegulaiity. The availability of uata that is
highly iegulai oi ieuunuant facilitates accuiate pieuiction of futuie events baseu on
past events. In one benchmaiking stuuy |1Sj, the effects of uata entiopy on accuiacy of
anomaly uetection weie evaluateu anu the following conclusions weie ieacheu: (1)
uiffeiences in uata iegulaiity uo influence the peifoimance of the anomaly uetectoi anu
(2) such uiffeiences can be founu in natuial enviionments. The piactical implication of
these conclusions is that peifoimance of anomaly uetection system cannot be evaluateu
by iunning it on uatasets of the same iegulaiity level, since theie is stiong eviuence to
suggest that iunning the same uetectoi on a uataset of uiffeient iegulaiity will geneiate
a uiffeient peifoimance inuex.


62

Shifting iegulaiities peisist acioss uomains, acioss netwoiks anu acioss
inuiviuual useis. Intiinsic stiuctuie of auuit uata useu by anomaly uetectoi affects its
peifoimance. 0ne possible solution is to monitoi level of iegulaiity in ieal time anu
swap cuiient anomaly uetectoi with anothei if it is expecteu to peifoim bettei foi new
iegulaiity level. Anothei solution is to use an auaptive anomaly uetectoi that
automatically calibiates its uetection paiameteis in iesponse to shifts in iegulaiity. 0ne
possible paiametei is the anomaly thiesholu that ueteimines the magnituue about
which the anomaly is taken seiiously (usually measuieu on u1 scale).
In oiuei to unueistanu the notion of uata entiopy, consiuei a simple auuit uata
sequence wheie each event is an action taken by a usei: system login (A), mail check
(B), open file spieausheet1.xls (C) anu logout (B). If a behavioi of a given usei is
examineu ovei a peiiou of uays, the following uata sequence might be geneiateu:
ABCBABCB Some events in such sequence necessaiily pieceue othei events. Such
sequential uepenuency between events in a categoiical uata is accounteu foi by a
measuie calleu conditionalrelativeentropy. Relativity ieflects the fact that entiopy uoes
not have uppei bounu, yet it still neeus to be measuieu on a fixeu scale. Conuitionality
ieflects the fact that both the piobability of event anu its pieuecessoi neeus to be
accounteu foi. Regulaiity inuex u signifies peifect iegulaiity (ieuunuancy) while
iegulaiity inuex 1 signifies peifect ianuomness.
The following types of anomalies weie intiouuceu into testing uata sets: foieign
symbol anomalies, foieign ngiam anomalies anu iaie ngiam anomalies.
Foreign symbol anomaly: this is the simplest type of anomaly. If the tiaining
uata uoes not contain a ceitain kinu of symbol (event), the occuiience of


6S

such event woulu constitute a foieign symbol anomaly. If the tiaining uata
containeu only symbols A, B, C anu B, symbol E in testing uata woulu be
consiueieu anomalous.
Foreign ngram anomaly: if the tiaining uata is geneiateu using an alphabet
of symbols (events) of size a, then theie aie o
n
possible ngiams of size n. If a
ceitain ngiam (sequence of events) uoes not appeai anywheie in the
tiaining uata, then the occuiience of such ngiam woulu constitute a foieign
ngiam anomaly. In the example given eailiei bigiam CB woulu be
consiueieu anomalous because event C always follows event B.
Rare ngram anomaly: ngiams that appeai infiequently in the tiaining uata
aie consiueieu iaie. Raiity is ueteimineu by a useispecifieu thiesholu.
When the anomaly uetectoi iuns on a uataset in tiaining moue, it cieates a piofile of
what is consiueieu noimal behavioi. The output of tiaining session consists of
specializeu uata stiuctuies neeueu to uetect anomalies. Specifically, it constiucts an
inteinal table that foi each unique ngiam stoies the piobability of its occuiience.
Buiing testing session, uetectoi is iun on uatasets containing ianuomly injecteu
anomalies. In oiuei to ueteimine the effects of uata iegulaiity on the uetectoi
peifoimance, both tiaining anu testing weie uone on uata of the same iegulaiity inuex.
The peifoimance of the uetectoi was evaluateu with iespect to accuiacy of uetection.
The expeiimental iesults |1Sj uemonstiateu a stiong coiielation between
iegulaiity levels anu falsealaim iate. In paiticulai, falsealaim iate incieases iapiuly as
the iegulaiity inuex giows. When iegulaiity inuex ieaches u.8, peifoimance is uegiaueu
significantly anu foi completely ianuom uata, false alaims ieach 1uu%. In the seconu


64

expeiiment, iegulaiities of systemcall stieams of S8 useis weie iecoiueu. Consiueiable
ueviations weie iuentifieu, pioving that uata iegulaiity is a ieallife phenomenon
beaiing both theoietical anu piactical significance foi the fielu of anomaly uetection.



6S

Chapter7
RelatedWorks

This chaptei piesents seveial technologies anu aieas of ieseaich that take the iuea of
infoimation secuiity into a new uiiection. It shoulu be noteu that these solutions aie
not alteinatives to intiusion uetection. Insteau they complement anu extenu NIBS in
new inteiesting ways.

7.1 Honeypots
Bistoiically, the majoi paiauigm of infoimation secuiity has been uefense. Intiusion
uetection systems, along with fiiewalls anu enciyption technologies have been
piimaiily focuseu on piotecting valuable assets fiom theft, coiiuption anu authoiizeu
access. As netwoiks giow in size anu complexity, the numbei of vulneiabilities
incieases exponentially anu so uoes the inciuence of attacks. Signatuie uetection
methous aie becoming less anu less effective uue to incieasing size of knowleuge
iepositoiies that neeu to be constantly upuateu anu maintaineu as well as polymoiphic
attacks, mentioneu in chaptei 1. Anomaly uetection engines aie becoming moie populai
uue to theii inheient ability to uetect novel attacks. But theie is an associateu cost of
high iate of false alaims.
Theie is a stiong inuication that the inciuence iate of attacks will continue to
inciease, while tools anu techniques useu by hackeis will giow in sophistication anu


66

vaiiability. It is becoming incieasingly uifficult foi secuiity piofessionals to keep up
with the enemy who has the uppei hanu. In many instances a new attack gets
uiscoveieu only aftei the uamage is alieauy uone anu the system has been
compiomiseu. Even if an anomaly engine manages to uetect anomalous tiaffic
associateu with the attack, the implementation of the iight iesponse policy is iaiely a
cleaicut uecision.
The assessment of counteimeasuies uepenus on the availability of auuitional
infoimation, such as the iuentity of attackeis, how they got in, how much uamage they
have causeu, etc. Since the attack is new, this infoimation is iaiely obtainable within a
shoit timefiame. The iesponse anu iecoveiy effoit is fuithei impeueu by the fact that
ceitain systems, even if compiomiseu, cannot be taken offline uue to the ciitical
functionality they pioviue. Foi example, the mail seivei of a laige oiganization is a
ciucial piouuction asset anu making it unavailable even foi a shoit peiiou of time might
cause moie uamage than attackeis evei hopeu foi.
Pait of the pioblem, as has been pieviously mentioneu, is that the stiategy foi
uealing with intiuueis has been piimaiily uefensive. 0nce the netwoik is aimeu with
fiiewalls, intiusion uetection systems, latest signatuies, tiaffic monitois, it enteis a
waiting state hoping that the uefenses withholu the attack long enough foi the iight
counteimeasuies to be ueviseu oi uetect it eaily enough so that no uamage is maue. A
technology, calleu honeypot attempts to change that.
Put simply, a honeypot is an infoimation system iesouice whose value lies in its
unauthoiizeu oi illicit use |2Sj. Boneypot is a uummy system with no piouuction value.
Theiefoie it sees no legitimate tiaffic anu any inteiaction with a honeypot is an


67

inuication of malicious activity. The puipose of honeypots is to luie hackeis into
attacking them with the intent of gatheiing infoimation. Eveiy action peifoimeu by the
intiuuei, fiom viewing files to inuiviuual keystiokes, can be loggeu anu latei analyzeu.
0nlike a piouuction system which potentially sees hunuieus of useis anu logs gigabytes
of uaily activity, honeypot collects small amount of veiy valuable infoimation. Since all
activity is malicious, theie is no noise anu uata analysis is both cheap anu easy.
Like all technologies, honeypots have associateu iisks. It shoulu alieauy be cleai
that honeypot loses its value as soon as the attackei who is inteiacting with it becomes
awaie of its piesence. The uangei howevei lies in the fact that some skilleu hackeis
coulu use a compiomiseu honeypot to lunch fuithei attacks against othei systems.
Seconu, theie is a iisk that once attackeis iuentify a honeypot, they will puiposefully
peifoim bogus actions in oiuei to misleau uata analysts, oi woise, they coulu locate
uata captuiing facilities anu uiiectly inject false infoimation. They coulu also coveitly
uisable specific paits of honeypot functionality.
Boneypots fall into two geneial categoiies: lowinteiaction honeypots anu high
inteiaction honeypots |2Sj. Lowinteraction honeypots emulate seivices anu opeiating
systems. Each emulateu seivice suppoits a subset of functionality that the actual seivice
allows. Foi example, emulateu FTP seivice may suppoit login anu only few othei
commanus. The main auvantage of lowinteiaction honeypot is that it is easy to ueploy
anu maintain. Also, since emulateu seivices allow limiteu functionality, the iisk of a
honeypot being tuineu into a hacking asset is substantially ieuuceu. The uisauvantages
incluue naiiow infoimation gatheiing capabilities, limiteu mainly to known attacks, anu
high piobability of exposuie.


68

Honeyd is most wiuely useu lowinteiaction honeypot system. It monitois the
space of unuseu IP auuiess on a netwoik, anu inteicepts anu ieuiiects any connection
attempt to such IP auuiess to itself, pietenuing to be a victim computei. It then caiefully
logs all the activity incluuing all inputs anu all issueu commanus. Honeyd can emulate
seivices as well as entiie opeiating systems (ovei Suu) |22j, both at the IP stack anu
application level. This way, if hackeis attempt to use 0S fingeipiinting tools, Honeyd
will issue a iesponse in accoiuance with the uesign of IP stack of the 0S it is emulating.
Fuitheimoie, Honeyd is not limiteu to emulating only uesktop opeiating systems. Foi
example, it coulu just as easily pietenu to be a Cisco ioutei. Honeyd is a veiy poweiful
tool that can be useu to builu an entiie netwoik of viitual uevices, each iunning its own
opeiating system anu seivices |2Sj.
Highinteraction honeypots involve ieal opeiating system anu seivices. Theie is
no emulation of any kinu. While ueployment anu maintenance of such systems is
without a uoubt costly enueavoi, the gains in powei outweigh the costs. Bigh
inteiaction honeypots allow attackeis to utilize full functionality of a system anu
theiefoie log extensive amount of infoimation. Bata collection is facilitateu by a special
activitylogging keinel mouule. This allows secuiity piofessionals to leain the full
extent of hackeis behavioi, iuentify new tools they use anu gain insight into theii
motives.

7.1.1 Honeynets
The concept of a highinteiaction honeypot coulu be extenueu to a honeynet |21j. A
honeynet is an entiie netwoik of honeypots. Within this netwoik all activity is caiefully


69

contiolleu anu loggeu. 0nce again, any incoming tiaffic into a honeynet is assumeu to be
malicious. At the heait of the honeynet aichitectuie is a honeywall, a gateway uevice
that sepaiate a honeynet fiom the iest of youi netwoik as well as the inteinet (See
figuie 1u). Boneywall is layei2 biiuge. Since biiuges aie tianspaient to piotocols
above the NAC layei, they aie invisible to anyone inteiacting with a honeynet.
Boneywall is iesponsible foi monitoiing anu logging all incoming tiaffic into a
honeynet. This uata is stoieu on an exteinal secuie system in oiuei to ensuie its not
uetecteu by hackeis who might make mouifications to it oi uelete it all togethei. Aftei
the uata has been collecteu, it neeus to be analyzeu anu conveiteu into applicable foim.

The final iesponsibility of a honeywall is Bata Contiol: isolation of a honeynet
fiom the iest of the netwoik. Eveiy effoit must be maue to ensuie that once the
attackeis entei the bounuaiy of the honeynet, they aie uenieu access to any outsiue

Internet
Productionnetwork
Honeynet
Honeywall
Figure 1u: Boneynet aichitectuie


7u

system. Bata Contiol acts as an intiusion pievention gateway anu fiiewall. ueneially,
the moie fieeuom attackeis aie alloweu to have within a honeynet, the moie can be
leaineu about them, anu the highei the iisk that Bata Contiol will be evaueu.
When useu foi ieseaich puiposes, honeypots collect infoimation on hackeis
activities, specific tools anu techniques they employ in oiuei to piobe netwoiks, exploit
vulneiabilities anu gain unauthoiizeu access to victim computeis. This infoimation
coulu then be useu to make systems moie secuie by fixing specific vulneiabilities that
alloweu hackeis to get in, ciaft new signatuies foi intiusion uetection systems oi
simply stuuy secuiity tienus. Nost impoitantly, honeypots allow secuiity piofessionals
to uevise iesponse stiategies against new attacks befoie they even occui on piouuction
systems.
Boneypots can also be useu by oiganizations to pioviue iealtime piotection
against intiusions. Netwoik scanning utilities piobe the taiget netwoik by scanning its
IP space foi live hosts anu then attempt to connect to open poits. By intiouucing a
honeypot that monitois unuseu IP space foi connection attempts, netwoik piobes can
be inteicepteu anu sloweu uown. Such honeypots aie known as sticky honeypots. As a
conciete example, LaBrea is a piogiam that cieates a viitual machine foi each unuseu
IP auuiess on a netwoik |2Sj. Each viitual machine waits foi a connection attempt anu
when one is maue, it uses vaiious TCP techniques to cause the senuei, in oui case a
netwoik piobe oi even a woim, to get stuck, sometimes foi a long time. 0ne stiategy
is to set winuow size of the iesponse packet to zeio, effectively placing senuei into a
holuing pattein as he waits foi the winuow size to inciease so he may senu uata.


71

Anothei way honeypots can piotect netwoiks fiom intiusions is by confusing
anu slowing uown human attackeis. While intiuueis aie wasting theii time inteiacting
with honeypots, it gives enough time to netwoik staff to iesponu anu ciicumvent the
attack. Boneypots aie an extiemely valuable secuiity asset. By captuiing small uata sets
of high value, they facilitate the uetection of new exploits anu even polymoiphic
shellcoue attacks uiscusseu in chaptei 1. In a way honeypots aie an ultimate foim of
anomaly uetection, because any activity within a honeypot is guaianteeu to be an
anomaly.

7.1.2 DynamicHoneypots
Netwoiks aie uynamic systems: new uevices aie auueu anu iemoveu, opeiating
systems aie upuateu, new applications anu seivices aie constantly intiouuceu. As the
netwoiks change, the honeypots themselves neeu to be mouifieu to ieflect the new
netwoik infiastiuctuie. 0utuateu honeypots quickly lose theii value as both
infoimationgatheiing anu intiusion pievention assets. Fuitheimoie, if honeypots anu
honeynets uont miiioi theii piouuction enviionment, they woulu stanu out too much
anu become easily iuentifiable. A tiauitional solution to this pioblem is manual
ieconfiguiation of honeypots by netwoik auministiative staff. This means time, money
anu unavoiuable mistakes.
A new pioposeu solution is a dynamichoneypot |22j that automatically analyzes
the netwoik on which it is ueployeu anu auapts to it. The iuea is to use iealtime passive
fingeipiinting to ueteimine which opeiating systems aie cuiiently useu in youi
netwoik, how many of each type theie aie, the IP iange on which each type of systems


72

iesiues on, anu which seivices they iun, anu then have a system like Honeyd miiioi
youi netwoik.

7.2 Honeycomb
ueneiating intiusion uetection signatuies is an aiuuous anu teuious task, geneially
iequiiing extensive knowleuge anu expeitise of secuiity piofessionals. As was
uiscusseu in chaptei 2, many uiffeient stanuaius exist foi tianslating an intiusion
scenaiio into a foimal signatuie. Since signatuie languages of vaiious NIBS uiffei in
both syntax anu level of expiessiveness, the signatuies cannot be easily poiteu fiom
one engine to be useu in the next. These issues can be auuiesseu by automatic signatuie
geneiation. We will use Honeycomb |12j as a case stuuy to uesciibe how netwoik
anomalies can be tiansfoimeu into signatuies iepiesenting vaiious attack scenaiios.
Honeycomb is a peifect example of how uiffeient secuiity technologies anu
concepts can extenu each otheis functionality in new inteiesting ways. Honeycomb is
implementeu as a honeyd plugin that has the following components: piotocol
analyzei, flow ieassemblei, pattein matchei anu signatuie geneiatoi. Netwoik tiaffic at
the honeyd honeypot is inspecteu at the netwoik anu tianspoit layeis. Foi each packet
flow, an empty signatuie is cieateu anu continually augmenteu with new facts. Flow
heie iefeis to a stieam of packets with the same souiceuestination IP anu poit.
Packet analyzei examines TCP, 0BP anu IP packet heaueis anu attempts to
uetect anomalies, foi example unusual TCP flag combinations. Each uiscoveieu anomaly
is iecoiueu as a new fact in a signatuie. The flow ieassemblei collects all packets


7S

belonging to the cuiient flow. Pattein matchei then attempts to finu similaiities in
packet payloaus between a cuiient connections flow anu those foi which connection
state is continually maintaineu. Each matcheu pattein is tieateu as a new fact anu
auueu to the signatuie.
Peiiouically, the signatuie pool is examineu anu uuplicate signatuies aie
uioppeu while ielateu signatuies aie aggiegateu. Signatuie geneiatoi tianslates native
signatuies into a specific foimat accepteu by NIBS. Cuiiently, Honeycomb geneiates
signatuies foi Bro anu Snort intiusion uetection systems. Initial testing showeu
piomising iesults: highquality signatuies weie geneiateu foi CodeRed II anu
Slammer woims anu vaiious poit scanneis |12j.

7.3 IdentifyingtheSourceofDoSAttacks
In chaptei 1, we uiscusseu how entiie netwoiks can be biought uown by floouing. We
also mentioneu that theie aie two typical ways of uealing with BoS attacks once they
have been uetecteu: iestoiing the netwoik opeiation by counteiing the effects of
floouing oi iuentifying the souice of the attack. Iuentifying the souice of BoS attack is a
lot moie uifficult that may initially seem since most BoS attacks use souice IP spoofing
anu uont leave much tiace. Bowevei, one ieseaich pioposes an inteiesting alteinate
solution |1uj.
The iuea is to iepiogiam all iouteis to stamp each packet with a special meta
uata that uniquely iuentifies the ioutei. This way upon ieceipt, one coulu extiact all
collecteu metauata anu theoietically tiace the packet all the way to its souice the
attackei himself. Such solution coulu put an enu to BoS attacks all togethei. If attackeis


74

knew that theie was no way to conceal theii iuentity, it woulu be unlikely foi them to
pioceeu with the attack in the fiist place.
Theie aie still seveial issues with the pioposeu solution which neeu to be
auuiesseu. The most intuitive way to iuentify a ioutei is by its inteinal IP auuiess. But if
all iouteis stampeu all packets with a S2bit IP auuiess, then by the time packets
ieacheu theii uestination, the metainfoimation they woulu accumulate along the way
woulu intiouuce a substantial tiansmission anu queuing uelays which woulu
unueimine the effective opeiation of the entiie inteinet.
The solution is to stoie only a poition of the entiie ioute insiue each packet.
Each ioutei woulu ianuomly ueciue baseu on some haiucoueu piobability whethei to
stamp a given packet with its IP auuiess as well as uistance fiom the souice of the
ioute. Luckily BoS attacks geneiate a substantial amount of tiaffic in oiuei to
accomplish theii goals, so chances aie that eveiy ioutei woulu stamp at least some
packet with this infoimation. At the uestination, a sequence of packets will be examineu
anu the entiie ioute will be ieconstiucteu piece by piece.
Even bettei solution is to stamp packets with smallei 16bit AS (autonomous
system) numbeis belonging to entiie iouting uomains, as oppose to inuiviuual iouteis.
Such numbeis coulu be easily stoieu insiue unuseu 16bit IP fiagmentation fielu.
Bowevei, skilleu hackeis coulu intentionally use fiagmentation fielu by fiagmenting
packets they senu into the netwoik. 0ne solution that coulu be iealizeu in the neai
futuie is to ieseive a ueuicateu 16bit fielu foi AS sampling in the upcoming IP piotocol
implementation (IPv6).



7S

Conclusion
Netwoiks aie becoming incieasingly complex as oiganizations auu new applications,
uevices anu useis. As the value of oui netwoik infiastiuctuie continues to giow, so
uoes oui neeu to ensuie that these assets aie auequately piotecteu against attacks that
piolifeiate in uiveisity, sophistication, speeu anu inciuence. We have ievieweu foui
types of systems: signatuie uetection systems, anomaly uetection systems, intiusion
iesponse systems anu honeypots. No single type of system has the potential to
effectively unueitake the task of netwoik suiveillance all by itself. Theie is an elevateu
neeu foi integiateu solutions that iip the benefits of both kinus of uetection paiauigms,
coupleu with a flexible autonomous iesponse element. The oveiheau of configuiing anu
maintaining multiple systems impeues this effoit especially in the context of netwoiks
that aie highly uynamic anu unpieuictable. Theiefoie we neeu extensible systems that
auapt to changing conuitions as well as inteiopeiate with othei systems to pioviue
complementaiy lines of uefense. We believe that the futuie of netwoik suiveillance will
be uiiven by uesign methouologies we have just iuentifieu: inteiopeiability,
auaptability, extensibility anu scalability.


76

References
|1j 0. Aikin. Netwoik Scanning Techniques 0nueistanuing Bow it is Bone.
PubliComCommunicationSolutions, Nov. 1999
|2j S. Axelsson. Intiusion uetection systems: A suivey anu taxonomy. Technical
Report, Chalmeis 0niv., Naich. 2uuu
|Sj B. }. Biown, B. Suckow, anu T. Wang. A Suivey of Intiusion Betection Systems.
Bepaitment of Computei Science, 0niveisity of Califoinia, San Biego
|4j B. Bebai. Intiouuction to IntiusionBetection Systems. IBMResearch, Zuiich
Reseaich Laboiatoiy.
|Sj B. Benning. An IntiusionBetection Nouel. IEEETrans.onSoftwareEng.,
Febiuaiy 1987
|6j L. Beii, S. Suin anu u. Naselli. Besign anu implementation of an anomaly
uetection system: An empiiical appioach. InProceedingsofTerenaTNC, 2uuS
|7j P. uaiciaTeouoio, }. E. Biazveiuejo, u. NaciFeinnuez, anu L. SnchezCasau.
Netwoikbaseu Bybiiu Intiusion Betection anu Boneysystems as Active
Reaction Schemes. IJCSNSInternationalJournalofComputerScienceandNetwork
Security, v0L.7 No.1u, 0ctobei 2uu7
|8j Bi. F. uong. Becipheiing Betection Techniques: Pait II AnomalyBaseu Intiusion
Betection |whitepapeij. McAfeeNetworkSecurityTechnologiesGroup, Naich
2uuS
|9j Bi. F. uong. Becipheiing Betection Techniques: Pait III Benial of Seivice
Betection |whitepapeij. McAfeeNetworkSecurityTechnologiesGroup, }anuaiy
2uuS


77

|1uj N. R. Bines. uoing Beyonu BehavioiBaseu Intiusion Betection. Bept. of
Computei Science, Binghamton 0niveisity, Fall 2uuS
|11j A. K. }ones anu R. S. Sielken. Computei System Intiusion Betection: A Suivey.
Bepaitment of Computei Science, 0niveisity of viiginia, 2uuu
|12j C. Kieibich, anu }. Ciowcioft. Boneycomb Cieating Intiusion Betection
Signatuies 0sing Boneypots. HotNetsII, Cambiiuge, 0SA, Novembei 2uuS
|1Sj S. Kumai anu E. B. Spaffoiu. A pattein matching mouel foi misuse intiusion
uetection. In17thNationalComputerSecurityConference, 1994.
|14j K. Leung anu C. Leckie. 0nsupeiviseu Anomaly Betection in Netwoik Intiusion
Betection 0sing Clusteis. Proc.28thAustralasianCSConf
|1Sj R. A. Naxion anu K. N. C. Tan. Benchmaiking anomalybaseu uetection systems.
InProceedingsof2000InternationalConferenceonDependableSystemsand
Networks
|16j Bi. P. Neumann anu P. Poiias. ENERALB: Event Nonitoiing Enabling Responses
to Anomalous Live Bistuibances. 1997NationalInformationSystemsSecurity
Conference, 0ctobei 1997
|17j K. Schafei. Intiusion Response The Right Couise of Action. 0niveisity of Iuaho
|18j R. Sekai, A. uupta, }. Fiullo, T. Shanbhag, S. Zhou, A. Tiwaii anu B. Yang.
Specificationbaseu Anomaly Betection: A New Appioach foi Betecting Netwoik
Intiusions. ACMCCS, 2uu2
|19j v. A. Siiis anu F. Papagalou. Application of anomaly uetection algoiithms foi
uetecting SYN floouing attacks. GlobalTelecommunicationsConference, 2uu4.
IEEE


78

|2uj Y. Song, N. E. Locasto, A. Staviou, A. B. Keiomytis anu S. }. Stolfo. 0n the
Infeasibility of Noueling Polymoiphic Shellcoue. InProceedingsofthe14thACM
ConferenceonComputerandCommunicationsSecurity (CCS). 0ctobei 2uu7,
Alexanuiia, viiginia, 0SA
|21j L. Spitznei. Know Youi Enemy: Boneynets. HoneynetProject, Nay 2uu6
|22j L. Spitznei. Bynamic Boneypots. SecurityFocus, Septembei 2uuS
|2Sj L. Spitznei. Boneypots: Befinitions anu values. Honeypots:TrackingHackers,
Nay 2uuS
|24j L. Spitznei. Know Youi Enemy: Passive Fingeipiinting. HoneynetProject, Naich
2uu2
|2Sj N. Stakhanova, S. Basu, }. Wong. A Taxonomy of Intiusion Response Systems.
Bepaitment of Computei Science, Iowa State 0niveisity
|26j K. Takemoii, K. Rikitake, Y. Niyake, anu K. Nakao. Intiusion Tiap System: An
Efficient Platfoim foi uatheiing Intiusionielateu Infoimation. Proceedingsofthe
10
th
InternationalConferenceonTelecommunications (ICT2uuS), IEEE, 2uuS
|27j N. Thottan anu C. }i. Anomaly Betection in IP Netwoiks. IEEETransactionsOn
SignalProcessing, vol. S1, No. 8, August 2uuS
|28j N. Roesch. Snoit Lightweight Intiusion Betection foi Netwoiks. Proc.USENIX
LISA'99Conf., Nov. 1999
|29j S. StanifoiuChen, S. Cheung, R. Ciawfoiu, N. Bilgei, }. Fiank, }. Boaglanu, K
Levitt, C. Wee, R. Yip, anu B. Zeikle. uiIBS A giaph baseu intiusion uetection
system foi laige netwoiks. InProceedingsofthe19thNationalInformation
SystemsSecurityConference, 1996


79

|Suj K2. ABNmutate REABNE. ABNmutate souice coue uistiibution. veision u.8.4.
0RL: http:www.ktwo.cacABNmutateu.8.4.tai.gz (}an 2uu2)
|S1j E. }. Bowuen. Netwoikbaseu intiusion uetection system buyeis guiue
|whitepapeij. EchoIdentitySystems,Inc.
|S2j L. A. uoiuon, N. P. Leob, W. Lucyshyn, anu R. Richaiuson. CSIFBI Computei
Ciime anu Secuiity Suivey, 2uuS.
|SSj N. Tanase. IP Spoofing: An Intiouuction. Secuiity Focus, Naich 11, 2uuS.
|S4j S. N. Specht anu R. B. Lee. Bistiibuteu Benial of Seivice: Taxonomies of Attacks,
Tools, anu Counteimeasuies. InProceedingsof17thInternationalConferenceon
ParallelandDistributedComputingSystems, 2004InternationalWorkshopon
Secuiity in Paiallel anu Bistiibuteu Systems. Septembei 2uu4.
|SSj u. Nnz, S. Li anu u. Caile. Tiaffic Anomaly Betection 0sing KNeans Clusteiing.
0niveisity of Tuebingen, ueimany, 2uu7.
|S6j C. Caivei, }. N. Bill, anu }. R. Suiuu. A methouology foi using intelligent agents to
pioviue automateu intiusion iesponse. InProceedingsoftheIEEESystems,Man,
andCyberneticsInformationAssuranceandSecurityWorkshop, 2uuu.
|S7j Bi. S. Fuinell anu N. Papauaki. Automateu Intiusion Response. Netwoik
Reseaich uioup, School of Computing, Communications & Electionics, 0niveisity
of Plymouth.
|S8j CERT. Ceit statistics: vulneiability iemeuiation. CERT Web Site, Apiil 2uu8.
http:www.ceit.oigstatsvulneiability_iemeuiation.html.