Flow Table Management

H.JonathanChao
ECEDepartment
chao@nyu.edu

EdgeNetworkManagement

ModifiedfromProf.Minlan Yusconferenceslides
atCoNEXT09andSigcomm10
2

EdgeNetworks
Enterprise networks
(corporate and campus)

Data centers (cloud)

Internet

Home networks
3

RedesignNetworksforManagement
Managementisimportant,yetunderexplored
Taking80%ofITbudget
Responsiblefor62%ofoutages

Makingmanagementeasier
Thenetworkshouldbetrulytransparent
Redesignthenetworks
tomakethemeasierandcheapertomanage
4

MainChallenges
FlexiblePolicies
(routing,security,
measurement)

LargeNetworks
(hosts,switches,apps)

Commodityswitches
(cost,energy,reliability)
5

LargeEnterpriseNetworks
Hosts
(10K 100K)

Switches
.
(1K 5K)

Applications
(100 1K)

LargeDataCenterNetworks
Switches
(1K 10K)

.
.

ServersandVirtualMachines
(100K 1M)
Applications
(100 1K)
7

FlexiblePolicies
Considerations:
Performance
Security
Mobility
Energysaving

Costreduction
Measurement
Debugging
Diagnosis
Maintenance

Alice

Customized Routing

Access Control

Alice
8

SwitchConstraints
Increasing linkspeed
(10Gbpsandmore)

Switch
Small,onchipmemory
(expensive,
powerhungry)

Scalingnetwork
Storinglotsofstate
Flexiblepolicies
Forwardingrulesformanyhosts/switches
AccesscontrolandQoS formanyapps/users
Monitoringcountersforspecificflows
9

TernaryContentAddressableMemory
(TCMA)
Enablestocompareadataagainstpredefinedsetofrulesinasingle
operation
Returnanaction(oraddress)associatedwiththefirstmatch
Eachruleconsistsofternarybits(0,1,ordontcare)
Commonusage:hardwarebasedpacketclassificationandflow
table
Comparingspecificheaderfields(e.g.,destinationaddress),against
rulesreflectingtheflowtable.

FromRamiCohenet.al.OntheeffectofforwardingtablesizeonSDNnetworkutilization

10

EdgeNetworkManagement
Specify
policies
ManagementSystem
Configure
devices

Collect
measurements

DIFANE[SIGCOMM10]
Scalingflexiblepolicydeployment
CAB[HotSDN14]
Enablingefficientrulecaching
11

ScalableflowbasednetworkingwithDIFANE
[Sigcomm10]
Minlan Yu,JenniferRexford,MichaelJ.Freedman,Jia Wang

12

TraditionalNetwork
Managementplane:
offline,sometimesmanual
Controlplane:
Hardtomanage
Dataplane:
Limitedpolicies

Newtrends:Flowbasedswitches&logicallycentralizedcontrol
13

Dataplane:FlowbasedSwitches
Performsimpleactionsbasedonrules
Rules:Matchonattributes inthepacketheader
Actions:Drop,forward,count
Storerulesinhighspeedmemory(TCAM)
Flow space
src. (X)

forwardvia
link1

TCAM (Ternary Content

Addressable Memory)

1.X:*Y:1 drop
2.X:5 Y:3 drop
3.X:1 Y:* count
4.X:*Y:* forward

dst.
(Y)

Countpackets

drop

14

ControlPlane:LogicallyCentralized
RCP[NSDI05],4D[CCR05],
Ethane[SIGCOMM07],
NOX[CCR08],Onix [OSDI10],
Softwaredefinednetworking

DIFANE:
Ascalablewaytoapply
finegrainedpolicies

15

PreinstallRulesinSwitches
Pre-install
rules

Packets hit
the rules

Controller

Forward

Problems:
TCAM space is limited (1000~4000 OpenFlow rules)
Nohostmobilitysupport
16

CacheRulesonDemand(Ethane)
Buffer and send
packet header
to the controller

Controller

cache
rules
First packet
misses the rules

Forward

Problems:
Computationloadatthecontrollerishigh
Delayofgoingthroughthecontroller
Switchesmisbehavewhenrequestingrules

17

DesignGoalsofDIFANE
Scalewithnetworkgrowth
LimitedTCAMatswitches
Limitedresourcesatthecontroller

Improveperpacketperformance
Alwayskeeppacketsinthedataplane

Minimalmodificationsinswitches
Nochangestodataplanehardware
Combinepreinstallationandcachingapproachesforbetterscalability
18

DIFANE:CombiningProactive&Reactive
Install
Prerules
install

Features
Host mobility

Cache
(Ethane)

DIFANE

Memory usage
Keep packet in
data plane
19

DIFANEArchitecture
(twostages)
DIstributed FlowArchitecture
forNetworkedEnterprises
DoingitFastandEasy

20

Stage1
Thecontrollerproactively generatesthe
rulesanddistributes themto
authorityswitches.

21

PartitionandDistributetheFlowRules
Controller

Distribute
partition
information Authority
Switch A

Flow space
AuthoritySwitch B

Authority
Switch C

Authority
SwitchB

Ingress
Switch

accept

Authority
SwitchA

reject

Egress
Switch

Authority
SwitchC

22

Stage2
Ingressswitchesreactively cacherules
fromauthorityswitches.

23

PacketRedirectionandRuleCaching
Authority
Switch
Ingress
Switch

Egress
Switch

First packet
Following
packets

Hit cached rules and forward

Aslightlylongerpathinthedata planeisfaster
thangoingthroughthe controlplane

24

LocateAuthoritySwitches
Partitioninformationiningressswitches
Usingasmallsetofcoarsegrainedwildcardrules
tolocatetheauthorityswitchforeachpacket

Adistributeddirectoryserviceofrules
Hashingdoesnot workforwildcards
Keyscanhavewildcardsinarbitrarybitpositions

AuthoritySwitchB
Authority
SwitchA

Authority
SwitchC

X:01Y:03 A
X:25Y:01B
X:25Y:23 C
25

PacketRedirectionandRuleCaching
R2

R2

R1

R1

Authority
Switch A
Ingress
Switch

Auth. Rules

Egress
Switch

First
packet
Cache Rules
PartitionRules

Following
packets

Hit cached rules and forward

Auth. Rules
R1

Go to
Go to
switch A switch B

Authority
Switch B

R1
26

ThreeSetsofRulesinTCAM
Type

Cache
Rules

Priority

Field1

Field 2

Action

Timeout

210

00**

111*

ForwardtoSwitchB

10sec

Iningressswitches
209
1110
11**
Drop
reactively installedbyauthorityswitches

10sec

110

00**

001*

Forward
Triggercachemanager

Infinity

Authority Inauthorityswitches
109
0001
0***
Drop,
proactively
installedbycontroller
Rules
Triggercachemanager

15

0***

000*

Redirecttoauth. switch

Partition Ineveryswitch
14

Rules
proactively installedbycontroller

27

DIFANESwitchPrototype
BuiltwithOpenFlow switch

Recv Cache
Updates
Control
Plane

Onlyin
Auth.
Switches

SendCache
Updates
Cache
Manager
Notification

CacheRules
Data
Justsoftwaremodificationforauthorityswitches
AuthorityRules
Plane
PartitionRules
28

CachingWildcardRules
Overlappingwildcardrules
Cannotsimplycachematchingrules
src.
dst.

Priority:
R1>R2>R3>R4

29

CachingWildcardRules
Multipleauthorityswitches
Containindependentsetsofrules
Avoidcacheconflictsiningressswitch
Authority
switch1

Authority
switch2
30

PartitionWildcardRules
Partitionrules
MinimizetheTCAMentriesinswitches
Decisiontreebasedrulepartitionalgorithm
CutBisbetter
thanCutA

CutB

CutA

31

HandlingNetworkDynamics
Network
dynamics

Cacherules

Authority
Rules

Partition
Rules

Policychanges
atcontroller

Timeout

Change

Mostly no
change

Topology
changesat
switches

Nochange

Nochange

Change

Hostmobility

Timeout

Nochange

Nochange
32

PrototypeEvaluation
Evaluationsetup
KernellevelClickbasedOpenFlowswitch
Trafficgenerators,switches,controllerrunon
separate3.0GHz64bitIntelXeonmachines

Comparedelayandthroughput
NOX:Bufferpacketsandreactivelyinstallrules
DIFANE:Forwardpacketstoauthorityswitches
33

Testbed forThroughputComparison
Testbed witharound40computers
Ethane

DIFANE

Controller

Controller

Authority
Switch

Traffic
generator
Ingress
switch

.
Traffic
generator

Ingress
switch

34

DelayEvaluation
Averagedelay(RTT)ofthefirstpacket
NOX:10ms
DIFANE:0.4ms

Reasonsforperformanceimprovement
Alwayskeeppacketsinthedataplane
Packetsaredeliveredwithoutwaitingforrulecaching
Easilyimplementedinhardwaretofurtherimproveperformance

35

PeakThroughput
Oneauthorityswitch;FirstPacketofeachflow
Throughput(flows/sec)

1,000K

1ingress 2 3 4
DIFANE
DIFANE
switch
DIFANE
Ethane
NOX
(800K)
100K
Ingressswitch
DIFANEisselfscaling:
Bottleneck
Higherthroughputwithmoreauthorityswitches.
(20K)
10K
Controller
Bottleneck(50K)

1K
1K
36

10K
100K
Sendingrate(flows/sec)

1000K

ScalingwithManyRules
Howmanyauthorityswitchesdoweneed?
Dependsontotalnumberofrules
andtheTCAMspaceintheseauthorityswitches
Campus

IPTV

#Rules

30K

5M

# Switches

1.7K

3K

Assumed Authority
SwitchTCAMsize
Required
#Authority Switches

160KB

1.6MB

5(0.3%)

100(3%)
37

Summary:DIFANEintheSweetSpot
Distributed

Logically-centralized

Traditionalnetwork
(Hardtomanage)

OpenFlow/Ethane
(Notscalable)

DIFANE:Scalablemanagement
Controllerisstillincharge
Switcheshostadistributed
directoryoftherules
38

CAB:AReactiveWildcardRuleCachingSystemfor
SoftwareDefinedNetworks
[HotSDN14]
Bo Yan, Yang Xu, Hongya Xing, Kang Xi, H. Jonathan Chao

39

 Reactively Caching Rules on

Demand

ReactivelyCachingRulesonDemand

Controller
Rule Set

Installonatdemand
a time
Install

Switch

10/23/2014

40

 Caching Wildcard Rules

CachingWildcardRules
Wildcard Rules enables:

Locality of Traffic

- Natural intention of
managing flows aggregately
- Less no. of rules stored.
Less invocations to controller
- Easy Update

10/23/2014

Update
!

NYC Dept.Edu (DoE) Data Center Traces

41

PrioritizedActionsforDifferentRules
Rule IPd

IPs

Prot. Port# Appl

Action

R1

128.238/16

TCP

telnet

Deny

R2

176.110/16

196.27.43/24

UDP

RTP

R3

196.27.43/24

134.65/16

TCP

R3: Drop if rate > 10 Mb/s

R1: Packet Filtering

R2: Policy Routing
R3: Traffic Policing

Send to port III

Drop if rate > 10 Mb/s

Net C

Net A

134.65/16

128.238/16
I

R1

IP network
Telnet
Router

II
Net B

R2
III

RTP

196.27.43/24

Net D
176.110/16
ATM network

42

 Challenge: Wildcard Rule Dependency

Challenge:WildcardRuleDependency

F2 (Dst IP)

Dependencyhas
chainreaction

f3

2
3

f1

f2

Rule set

F1 (Src IP)

Wrong
matching!

f1

f3

f2

f1

f3

Hypothetical
10/23/2014

Caching
100s dependent
rules

f2

Switch Mem
90%

43

 Methods to Accommodate Rule

MethodstoAccommodateRuleDependency
Dependency
Cache all dependent rules
- Increase memory use for each flow

Cache exact match rules

f2

Split rule set and cache micro rules

- Generates significantly more rules

F2 (Dst IP)

- Leads to frequent rule installation (per-flow)

f3
f1
F1 (Src IP)

Switch Mem

Inefficient rule management increases switch memory use causes

- more cache miss at switch

- higher controller load, control bandwidth
- longer flow setup delay
Problem:howtoaccommodateruledependencywithefficientmemuse?
10/23/2014

 Solution: CAching
rules in Buckets (CAB)
Solution:CAching
rulesinBuckets(CAB)
F1
Buckets

Cache Miss

f3
f2
f1

Rules

Buckets

BucketC
BucketF

Associate
Rules
Rule2
Rule3
Rule4

OpenFlow Switch

F2

CAB Controller

B 3

f2

5
9

f1 F

RuleSwitch
set (Controller)
Mem

Matched
Bucket
C, andFilter
Rule 3
Cache Miss
at Bucket
Now no
100s
rules
up more
fInstall
C and
and
ruledependent
&4
A
rule
23 (&3)
2 is setbucket
Only rules within requested bucket
f31 is set up
10/23/2014

f3

45

 Bucket size affects memory

Bucketsizeaffectsmemoryefficiency
efficiency

F2

F1

f3 D

f2

5
9

f1

BI
M

2x2 buckets
More rules cached each time
Unmatched rules cached

4x4 buckets
More buckets cached

Choosing bucket size affects switch memory efficiency

10/23/2014

46

 Bucket Generation Decision Tree

BucketGenerationDecisionTree
Decision tree based generation algorithm [HyperCut]
Partition till the no. of associate rules in each bucket is bounded
Bucket size

F1

Whole Field
Space

F2

2
1

10/23/2014

Bucket B
R1 R3

Partition on F2
Bucket A
R2 R3

4
B

Partition on F1

2
Bucket C
R3 R4

Technical problems:
How to select the fields to partition?
[see paper]
47

 Preliminary Evaluation Setup

EvaluationSetup
Generate synthetic rule set using [Classbench]
Map the headers of NYCDoE traces to the synthetic rules
Test different caching schemes
Real traces
from DC

Caching
Schemes

Headermapping
Synthetic
rules by
ClassBench

10/23/2014

Trace
generator

Synthetic
traces

Rule Caching
Simulator

48

 Preliminary Evaluation Setup

EvaluationSetup
Performance Evaluation
Cache miss rate
Bandwidth consumption
Flow setup latency (see paper)

Parameter Setting
TCAM capacity is set to support 1500 entries
Effects of tuning bucket size

Comparison

10/23/2014

CAching rules in Buckets (CAB)

Caching exact match rules (CEM)
Caching micro rules (CMR)
Caching dependent rules (CDR)

49

CachemissandControlbandwidthusage
Memory overflow

CDR: Dependent Rules

CEM: Exact Match

CMR: Micro Rules

> half less control bandwidth use

> 10x less cache miss

CAB: Bucket + Rules

10/23/2014

50

Effectoftuningbucketsize
Memory overflow

Small bucket
10/23/2014

Large bucket
51

Summary
CAB is a novel wildcard rule caching system which
Resolves rule dependency in wildcard rule caching
Achieves efficient switch memory use and helps reducing
control network bandwidth
flow setup latency
controller load
Fully compatible with the latest OpenFlow standards

10/23/2014