OpenStack networking - with Open vSwitch VLAN, GRE

Paul Sim Cloud Consultant paul.sim@canonical.com

Index Prior Knowledge OpenStack Networking - VLAN OpenStack Networking - GRE Security Group, Floating-IP, NameSpace Neutron ML2

Prior Knowledge - Network NameSpace

without Network NameSpace
Process Process Process Process Process Process Process Share
BMW NameSpace Network Resources Ford NameSpace Network Resources Benz NameSpace Network Resources

with Network NameSpace

Process

Network Resources
Routing table Address Netfilter rules eth0 eth1 eth2

Network Resources
eth0 eth1 eth2

Network NameSpace provides isolation of the system resources associated with networking. Thus, each network namespace has its own network devices, IP addresses, IP routing tables, /proc/net directory, port numbers, and so on. - http://lwn.net/Articles/531114/

Prior Knowledge - VLAN, GRE

VLAN - Virtual LAN

802.1Q Header TPIC : 16bit - 0x8100 TCI : 16bit PCP : 3bit DEI : 1bit VID : 12bit (0 ~ 4095)

GRE - Generic Routing Encapsulation

16 Bytes Header + IP header Key field : 32bit - identify an individual traffic flow within a tunnel

OpenStack Installation - Grizzly

External network 192.168.122.0/24

eth0

eth0

eth0

eth0

Controller node
Nova Keystone

Network node
Quantum L3-agent Quantum openvswitch-agent Quantum metadata-agent Quantum dhcpagent eth1 eth2

Compute node - 1
Quantum openvswitch-agent Nova compute

Compute node - 2
Quantum openvswitch-agent Nova compute

Glance

Horizon

Quantum - Server eth1 eth2

eth1

eth2

eth1

eth2

Management 192.168.20.0/24 Data 192.168.10.0/24

Network Topology

ext_net : external network - 192.168.122.0/24 net_proj_one : user_one tenant - 50.50.1.0/24 net_proj_two : user_one tenant - 50.50.2.0/24 net_proj_new : user_new tenant - 60.60.1.0/24

Big picture - VLAN

OpenStack Grizzly OpenvSwitch plug-in VLAN mode

Network node
net_proj_one net_proj_two net_proj_new

Compute node - 1
VM VM tap~ tag:2 VM tap~ tag:2

tap~

tap~

tap~

qr~

qr~

qr~

tap~ tag: 1
int-br-eth1

br-int
qg~ qg~ qg~

int-br-eth1 Data 192.168.10.0 /24

br-int

phy-br-eth1

phy-br-eth1

br-ex
eth0

br-eth1

eth1

eth1

br-eth1

OVS port OVS Bridge

qg~~~ : external gateway interface qr~~~ : virtual router interface

VLAN - Compute node

OpenStack Grizzly OpenvSwitch plug-in VLAN mode

Compute node - 1

VM

VM tap~ tag:2

VM tap~ tag:2

VM tap~ tag:3

br-eth1

eth1

tap~ tag: 1
veth pair int-br-eth1

phy-br-eth1

br-int

Packet conversion
mod_vlan_vid mod_vlan_vid

Security Group[1]

VLAN - Compute node

Packet conversion
janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-eth1 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=90455.716s, table=0, n_packets=6, n_bytes=468, priority=2,in_port=2 actions=drop cookie=0x0, duration=89606.096s, table=0, n_packets=9484, n_bytes=2312018, priority=4,in_port=2,dl_vlan=1 actions=mod_vlan_vid:1024,NORMAL cookie=0x0, duration=90456.248s, table=0, n_packets=6813, n_bytes=1325511, priority=1 actions=NORMAL janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=90458.482s, table=0, n_packets=64, n_bytes=4644, priority=2,in_port=1 actions=drop cookie=0x0, duration=89608.755s, table=0, n_packets=6499, n_bytes=1283680, priority=3,in_port=1,dl_vlan=1024 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=90459.075s, table=0, n_packets=9820, n_bytes=2323195, priority=1 actions=NORMAL

openvswitch-agent.log
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-int', 'hard_timeout=0, idle_timeout=0,priority=3,in_port=1,dl_vlan=1024,actions=mod_vl an_vid:1,normal'] Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-eth1', 'hard_timeout=0, idle_timeout=0,priority=4,in_port=2,dl_vlan=1,actions=mod_vlan _vid:1024,normal']

VLAN - Network node

OpenStack Grizzly OpenvSwitch plug-in VLAN mode Network node
tap~ tap~ tap~

Namespcae

Namespcae

Namespcae

qr~ qg~ qg~

qr~ qg~

qr~

br-eth1

eth1

veth pair

br-int br-ex
eth0
net_proj_one net_proj_two

int-br-eth1

phy-br-eth1

Packet conversion
mod_vlan_id

Floating-IP(NAT)
net_proj_new

mod_vlan_id

VLAN - Network node

Packet conversion
janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=7370.307s, table=0, n_packets=6, n_bytes=468, priority=2,in_port=6 actions=drop cookie=0x0, duration=7368.424s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=6,dl_vlan=2048 actions=mod_vlan_vid:2,NORMAL cookie=0x0, duration=7367.991s, table=0, n_packets=764, n_bytes=191460, priority=3,in_port=6,dl_vlan=1024 actions=mod_vlan_vid:3, NORMAL cookie=0x0, duration=7369.073s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=6,dl_vlan=500 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=7370.924s, table=0, n_packets=549, n_bytes=104066, priority=1 actions=NORMAL janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-eth1 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=7373.826s, table=0, n_packets=14, n_bytes=1104, priority=2,in_port=2 actions=drop cookie=0x0, duration=7372.725s, table=0, n_packets=13, n_bytes=922, priority=4,in_port=2,dl_vlan=1 actions=mod_vlan_vid:500,NORMAL cookie=0x0, duration=7371.663s, table=0, n_packets=519, n_bytes=103966, priority=4,in_port=2,dl_vlan=3 actions=mod_vlan_vid:1024, NORMAL cookie=0x0, duration=7372.09s, table=0, n_packets=9, n_bytes=634, priority=4,in_port=2,dl_vlan=2 actions=mod_vlan_vid:2048,NORMAL cookie=0x0, duration=7374.384s, table=0, n_packets=764, n_bytes=191460, priority=1 actions=NORMAL

Big picture - GRE

OpenStack Grizzly OpenvSwitch plug-in GRE tunneling

Network node
net_proj_one net_proj_two net_proj_new

Compute node - 1

tap~

tap~

tap~

Data 192.168.10.0 /24

VM

VM tap~ tag:2

br-tun

qr~

qr~

qr~ patch

Tunnel

br-tun

br-int

patch

gre~

tap~ tag: 1
patch

gre~

patch

br-int

qg~

qg~

qg~

br-ex
eth0
OVS port OVS Bridge

qg~~~ : external gateway interface qr~~~ : virtual router interface

GRE - Compute node

OpenStack Grizzly OpenvSwitch plug-in GRE tunneling

Compute node - 1

VM

VM tap~ tag:2

VM tap~ tag:2

VM tap~ tag:3

br-tun

gre~

Tunnel

tap~ tag: 1

patch

patch

br-int

Packet conversion
mod_vlan_vid set_tunnel id

Security Group[1]

GRE - Compute node

Packet conversion
janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0x0, duration=87770.027s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00: 00 actions=mod_vlan_vid:1,output:1 cookie=0x0, duration=87770.09s, table=0, n_packets=8786, n_bytes=1893724, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL cookie=0x0, duration=87769.693s, table=0, n_packets=3031, n_bytes=617650, priority=3,tun_id=0x1,dl_dst=fa:16:3e:db:08:63 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=87769.966s, table=0, n_packets=6320, n_bytes=4432680, priority=3,tun_id=0x1,dl_dst=fa:16:3e:e0:73:95 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=87771.753s, table=0, n_packets=2921, n_bytes=951454, priority=1 actions=drop

GRE - Network node

OpenStack Grizzly OpenvSwitch plug-in GRE tunneling Network node
tap~ tap~ tap~

Namespcae

Namespcae

Namespcae

qr~ qg~ qg~

qr~ qg~

qr~

Tunnel gre~

br-tun
patch

br-int br-ex
eth0
net_proj_one net_proj_two

patch

Packet conversion
set_tunnel id

Floating-IP(NAT)
net_proj_new

mod_vlan_id

GRE - Network node

Packet conversion
janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0x0, duration=474674.446s, table=0, n_packets=7899, n_bytes=2572502, priority=3,tun_id=0x3,dl_dst=01:00:00:00:00:00/01:00:00:00:00: 00 actions=mod_vlan_vid:2,output:1 cookie=0x0, duration=473163.123s, table=0, n_packets=7876, n_bytes=2565284, priority=3,tun_id=0x4,dl_dst=01:00:00:00:00:00/01:00:00:00:00: 00 actions=mod_vlan_vid:3,output:1 cookie=0x0, duration=633937.826s, table=0, n_packets=10543, n_bytes=3426814, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00: 00 actions=mod_vlan_vid:1,output:1 cookie=0x0, duration=473163.329s, table=0, n_packets=16484, n_bytes=3348666, priority=4,in_port=1,dl_vlan=3 actions=set_tunnel:0x4, NORMAL cookie=0x0, duration=474674.541s, table=0, n_packets=16864, n_bytes=3389132, priority=4,in_port=1,dl_vlan=2 actions=set_tunnel:0x3, NORMAL cookie=0x0, duration=633937.905s, table=0, n_packets=62044, n_bytes=37320316, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1, NORMAL cookie=0x0, duration=472911.069s, table=0, n_packets=16335, n_bytes=3551350, priority=3,tun_id=0x4,dl_dst=fa:16:3e:89:fd:ce actions=mod_vlan_vid:3,NORMAL cookie=0x0, duration=474336.184s, table=0, n_packets=16360, n_bytes=3560332, priority=3,tun_id=0x3,dl_dst=fa:16:3e:d8:d5:29 actions=mod_vlan_vid:2,NORMAL cookie=0x0, duration=474674.351s, table=0, n_packets=525, n_bytes=52427, priority=3,tun_id=0x3,dl_dst=fa:16:3e:69:ca:97 actions=mod_vlan_vid:2,NORMAL cookie=0x0, duration=473162.912s, table=0, n_packets=197, n_bytes=19365, priority=3,tun_id=0x4,dl_dst=fa:16:3e:d6:b8:07 actions=mod_vlan_vid:3,NORMAL cookie=0x0, duration=633937.746s, table=0, n_packets=6207, n_bytes=630043, priority=3,tun_id=0x1,dl_dst=fa:16:3e:c7:ec:bd actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=474794.912s, table=0, n_packets=36912, n_bytes=7440964, priority=3,tun_id=0x1,dl_dst=fa:16:3e:8b:a6:d7 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=636252.069s, table=0, n_packets=163, n_bytes=36046, priority=1 actions=drop

Security Group - VLAN, GRE

FORWARD

quantum-filter-top quantum-openvswi-local Security group is applied here quantum-openvswi-FORWARD quantum-openvswi-sg-chain quantum-openvswi-iTAP_NUMBER quantum-openvswi-sg-fallback quantum-openvswi-oTAP_NUMBER quantum-openvswi-sg-fallback

Security Group - VLAN, GRE

Chain quantum-openvswi-sg-chain (4 references) target prot opt source destination quantum-openvswi-i21767f1f-4 all -- 0.0.0.0/0 0.0.0.0/0 quantum-openvswi-o21767f1f-4 all -- 0.0.0.0/0 0.0.0.0/0 quantum-openvswi-i7903fd30-7 all -- 0.0.0.0/0 0.0.0.0/0 quantum-openvswi-o7903fd30-7 all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap21767f1f-45 --physdev-is-bridged PHYSDEV match --physdev-in tap21767f1f-45 --physdev-is-bridged PHYSDEV match --physdev-out tap7903fd30-74 --physdev-is-bridged PHYSDEV match --physdev-in tap7903fd30-74 --physdev-is-bridged

Chain quantum-openvswi-i7903fd30-7 (1 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 RETURN udp -- 50.50.1.3 0.0.0.0/0 udp spt:67 dpt:68 quantum-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 Chain quantum-openvswi-o7903fd30-7 (2 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 MAC ! FA:16:3E:DB:08:63 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 DROP all -- !50.50.1.2 0.0.0.0/0 DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED RETURN all -- 0.0.0.0/0 0.0.0.0/0 quantum-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0

[1] Note, OpenStack uses iptables rules on the TAP devices such as tap~~ to implement security groups,. However, Open vSwitch is not compatible with iptables rules that are applied directly on TAP devices that are connected to an Open vSwitch port.

Network NameSpace

janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 ifconfig lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 qg-fa243f49-d6 Link encap:Ethernet HWaddr fa:16:3e:9f:4b:63 inet addr:192.168.122.50 Bcast:192.168.122.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe9f:4b63/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 qr-bc654dc2-f1 Link encap:Ethernet HWaddr fa:16:3e:c7:ec:bd inet addr:50.50.1.1 Bcast:50.50.1.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fec7:ecbd/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 192.168.122.1 0.0.0.0 UG 0 0 0 qg-fa243f49-d6 50.50.1.0 * 255.255.255.0 U 0 0 0 qr-bc654dc2-f1 192.168.122.0 * 255.255.255.0 U 0 0 0 qg-fa243f49-d6

Floating-IP(NAT) - VLAN, GRE

NameSpace
janghoon@Network-node:~$ sudo ip netns show qdhcp-4c2f2346-ffaa-41a0-ab76-34cadf0163f5 qrouter-e1b88ce4-51e9-4744-be80-d70d04c6a59b qdhcp-c19e22a0-1700-4b3b-91e5-2c961ef0a353 qrouter-244fff3f-f935-4bdd-949d-739f1ce81dd0 qdhcp-f37b681a-4be8-47b8-8063-3d17d24ee1ae qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0

Floating-IP(NAT)
janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 iptables -L -n -t nat Chain quantum-l3-agent-PREROUTING (1 references) target prot opt source destination REDIRECT tcp -- 0.0.0.0/0 169.254.169.254 tcp dpt:80 redir ports 9697 DNAT all -- 0.0.0.0/0 192.168.122.51 to:50.50.1.2 Chain quantum-l3-agent-float-snat (1 references) target prot opt source destination SNAT all -- 50.50.1.2 0.0.0.0/0 to:192.168.122.51 Chain quantum-l3-agent-snat (1 references) target prot opt source destination quantum-l3-agent-float-snat all -- 0.0.0.0/0 SNAT all -- 50.50.1.0/24 0.0.0.0/0

0.0.0.0/0 to:192.168.122.50

Neutron ML2
The Modular Layer 2 (ML2) plugin is a framework allowing OpenStack Networking to simultaneously utilize the variety of layer 2 networking technologies found in complex real-world data centers. It currently works with the existing openvswitch, linuxbridge, and hyperv L2 agents, and is intended to replace and deprecate the monolithic plugins associated with those L2 agents.

Neutron
ML2 Plugin
TypeDriver
OpenvSwitch

MechanismDriver
Arista Cisco Nexus OpenDaylight

Hyper-V

VLAN

GRE

VxLAN

Flat

pSwitch

TypeDriver : TypeDrivers maintain any needed type-specific network state, and perform provider network validation and tenant network allocation. MechanismDriver : The MechanismDriver is responsible for taking the information established by the TypeDriver and ensuring that it is properly applied given the specific networking mechanisms that have been enabled. https://wiki.openstack.org/wiki/Neutron/ML2

Neutron ML2

eth0

eth0

eth0

Network node
Neutron L3-agent Neutron ML2 plugin Neutron metadataagent Neutron dhcpagent eth1 eth2

Compute node - 1
Neutron ML2-agent Nova compute

Compute node - 2
Neutron ML2-agent Nova compute

eth1

eth2

eth1

eth2