BRKDCN 3040

Troubleshooting
VxLAN BGP EVPN
Vinit Jain - CCIE# 22854

BRKDCN-3040
@vinugenie
Cisco Spark
Questions?
Use Cisco Spark to chat with the
speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
Cisco Spark spaces will be cs.co/ciscolivebot#BRKDCN-3040

available until July 3, 2017.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
• VxLan Overview • Troubleshooting VxLAN EVPN

• Flood & Learn Mechanism Data Plane
• Intro to VxLan BGP EVPN • VxLAN OAM
• Configuration • Case Studies
• Troubleshooting VxLAN EVPN
Control Plane
• Route Advertisement and Installation,
L2/L3 VNI
VxLAN & VxLAN BGP
EVPN Overview
VxLAN Overview
Simple Definition
• VXLAN Overlay
- Layer 2 overlay on top of your Layer 3 underlay
• VxLAN Network Identifier
− Each VxLAN segment is identified by a unique 24-bit segment ID
− Only hosts on the same VNI are allowed to communicate with each other
• Benefits
− Overcome 4094 VLAN Scale limitation
− Better utilization of available network paths
− Multi-Tenant with virtualization
BRKDCN-3040 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
VxLAN Overview
Overview
Edge Device
Edge Device
Local LAN
Local LAN Segment
IP Interface
Segment
Physical
Host Physical
Edge Device
Local LAN Host
Segment
Virtual Switch
Virtual Hosts
VxLAN Overview
Overview
VTEP
VTEP
V V
Local LAN
Local LAN Segment
Segment Encapsulation
Physical
Host VTEP
V Physical
Local LAN Host
Segment
Virtual Switch
VTEP – VXLAN Tunnel End-Point

VNI/VNID – VXLAN Network Identifier
Virtual Hosts
VxLAN Overview
VXLAN Concepts
• VXLAN Overlay
- A VXLAN Overlay or VXLAN segment is a Layer-2 broadcast domain identified by the
VNID that extends or tunnels traffic from one VTEP to another.
• VXLAN Tunnel End Point (VTEP)
- A VTEP is a device that provides both encapsulation and de-capsulation of classical
Ethernet and VXLAN packets to and from a VXLAN segment
- Each VTEP may have the following types of interfaces:
 Switchport interfaces on the local LAN segment to support local endpoints
 Layer-3 interfaces to the transport IP network
 SVI interfaces
• VXLAN Gateway
- A VTEP that bridges traffic between VXLAN segments
VXLAN Encapsulation
Ethernet
Payload FCS
Header
Outer Outer Inner

Outer IP VXLAN Payload New FCS
Ethernet UDP Ethernet
Flags Reserved Instance ID Reserved
8 Bytes 24 Bytes 24 Bytes 8 Bytes
Rsvd Rsvd
Outer UDP Destination Port = VXLAN (originally 8472, recently updated to 4789)
1
Outer UDP Source Port = Hash of Inner Frame Headers (optional)
• The outer IP header has the source IP and destination IP of the VTEP endpoints
• The outer Ethernet header has the source MAC of the source VTEP and the
destination MAC of the immediate Layer-3 next hop
VxLAN Packet Structure
F
Outer MAC Outer UDP VxLAN
Outer IP Header Original Layer 2 Frame C
Header Header Header
S
14 Bytes
(4 bytes optional) 8 Bytes Hash of inner L2/l3/L4 headers
Destination NH MAC Source Port
of original frame. Enables
16 entropy for ECMP Load
48 Address Addr
16 VxLAN Port UDP 4789 balancing
48 Source Address
16 UDP Length
16 VLAN Type
0x8100 16 Checksum
16 VLAN ID Tag
Src and Dst
16 Ether Type 20 Bytes Addresses of 8 Bytes
0x0800
VTEP 16M possible
72 IP Header Misc. VxLAN
Data 8 RRRR1RRR segments
8 Protocol 0x11 Reserved
Src VTEP MAC Addr 24
16 Header
24 VNI
32 Checksum
Source IP 8 Reserved
32
Destination IP
VxLAN Overview
VxLAN Gateway Types
• Layer 2 Gateway
− The layer 2 gateway is required when the layer 2 traffic (IEEE 802.1q tagged traffic)
comes from VLAN into VxLAN segment (encapsulation) or
− The Ingress VxLAN packet egresses out an 802.1q tagged interface (de-encapsulation),
where the packet is bridged to a new VLAN.
• Layer 3 Gateway
− A layer 3 gateway is used when there is a VxLAN to VxLAN routing
− The ingress packet is a VxLAN packet on a routed segment but the packet egresses out
on a tagged 802.1q interface and the packet is routed to a new VLAN
VxLAN Overview
VXLAN Gateway Types Ingress VXLAN packet
Egress packet is IEEE 802.1q
tagged interface. packet is
on RED segment
BRIDGED to new VLAN
• VXLAN to VLAN Bridging V

• (Layer-2 Gateway) VXLAN Layer-
2 Gateway
Ingress VXLAN packet Egress VXLAN packet is ROUTED

on RED segment to new VXLAN segment
• VXLAN-to-VXLAN Routing
• (Layer-3 Gateway) V
VXLAN Router
• VXLAN-to-VLAN Routing Ingress VXLAN packet

Egress packet is IEEE 802.1q
tagged interface. packet is
on RED segment ROUTED to new VLAN
• (Layer-3 Gateway)
V
VXLAN Router
VxLAN Overview
MTU
• VxLAN adds 50 bytes to the original Ethernet frame.
• VTEPs MUST NOT fragment VXLAN packets.
• Intermediate routers may fragment encapsulated VXLAN packets due to the
larger frame size.
• The destination VTEP MAY silently discard such VXLAN fragments.
• To ensure end-to-end traffic delivery without fragmentation, it is
RECOMMENDED that the MTUs (Maximum Transmission Units) across the
physical network infrastructure be set to a value that accommodates the larger
frame size due to the encapsulation.
VxLAN Overview
Segment ID
Segment IDs are utilized for providing isolation at

Layer 2 and Layer 3 across the Fabric.
802.1Q tagged frames received at the Leaf nodes

from edge devices must be mapped to specific
Segments.
The VLAN to Segment mapping is performed on a

leaf device level VLANs become locally significant
on the leaf node and 1:1 mapped to a Segment ID.
Thus, Segment IDs are globally significant and

VLAN IDs are locally significant.
End System End System
VxLAN Overview 3
ARP Request for IP B
VxLAN - Flood and Learn Src MAC: MAC-A
Dst MAC: FF:FF:FF:FF:FF:FF
IP-3
VTEP 3
S-MAC: MAC-1
D-MAC: MAC VxLAN Remote
VTEP-3
00:01:5E:01:01:01 Address ID VTEP
Outer S-IP: 192.168.1.1 MAC-A 10 192.168.1.1

4
Outer D-IP: 239.1.1.1
UDP ARP Response from IP B
Src MAC: MAC-B
VXLAN VNID: 10
Dst MAC: MAC-A
7
2
Src MAC: MAC-A
Dst MAC: VTEP 2
ARP Response from IP B Mcast Group
FF:FF:FF:FF:FF:FF IP-2 End System B
Src MAC: MAC-B 239.1.1.1
MAC-2 MAC-B
Dst MAC: MAC-A IP - B
2
VTEP-1 VTEP-2 3
End System A S-MAC: MAC-2
MAC-A D-MAC: MAC-1 ARP Request for IP B
IP - A 5 Src MAC: MAC-A
VTEP 1
1 IP-192.168.1.1 Outer S-IP: 192.168.2.2 Dst MAC: FF:FF:FF:FF:FF:FF
6
MAC-1 Outer D-IP: 192.168.1.1
MAC VxLAN Remote UDP MAC VxLAN Remote
Address ID VTEP Address ID VTEP
Src MAC: MAC-A
Dst MAC: FF:FF:FF:FF:FF:FF VXLAN VNID: 10
MAC-B 10 IP-2 MAC-A 10 192.168.1.1
ARP Response
VxLAN Overview
VxLAN – Flood and Learn
• Data Plane learning technique for VxLAN
• VNI’s are mapped to a multicast group on a VTEP
• Local MACs are learnt over a VLAN (VNI) on a VTEP
• Broadcast, Unknown Unicast, Multicast (BUM Traffic) is flooded to the delivery
multicast group for that VNI
• Remote VTEPs part of same multicast group learn host MAC, VNI and source
VTEP as the next-hop for the host MAC from flooded traffic
• Unicast packets to the host MAC are sent directly to source VTEP as VxLAN
encapsulated packet
VxLAN Overview
Ingress Replication
• Some customers not comfortable deploying multicast in their core
• With Ingress Replication (IR), BUM traffic ingress access side is replicated to
remote VTEP as unicast
• Static IR VETP tunnel is kept alive as long as the route to the VTEP is available.
• Support multiple VTEPs per VNI and a VTEP in multiple VNIs
• Up to 16 static IR VTEPs recommended
• Multicast and IR config can co-exist on the same switch nodes but on different
VNI’s
BGP for VxLAN
WHY?
• Control plane learning for end host Layer-2 and Layer-3 reachability
information to build more robust and scalable VXLAN overlay networks.
• Leverages the decade-long MP-BGP VPN technology to support scalable

multi-tenant VXLAN overlay networks.
• EVPN address family carries both Layer 2 and Layer 3 reachability

information. This provides integrated bridging and routing in VXLAN overlay
networks.
VxLAN Overview
VXLAN EVPN VNI Types
Tenant 1 (VRF 1)
SVIX SVIA SVI

B
Layer-3 VNI X’ Layer-2 VNI A’ Layer-2 VNI B’
VLAN X VLAN A VLAN B
• 1 Layer-3 VNI per • 1 Layer-2 VNI per Layer-2 segment

Tenant (VRF) for • Multiple Layer-2 VNIs per tenant
routing
• VNI A’ and B’ are used for bridged
• VNI X’ is used for packets
routed packets
VxLAN Overview
VxLAN EVPN
BGP-EVPN BGP-EVPN
Mac_H1: VNI1 Mac_H3: VNI1

IP_H1: VNI3 IP_H3: VNI3
NHOP: VTEP-1 NHOP: VTEP-1
IP Multicast Core
Mac_H2: VNI2 Mac_H4: VNI2
IP_H2: VNI3 IP_H4: VNI3
VTEP-1 VTEP-2
NHOP: VTEP-1 NHOP: VTEP-2
V V
VNI3 VRF VRF VNI3
Vlan-500 A A Vlan-500
L3 VNI L3 VNI
MAC_H1 MAC_H1 MAC_H3 MAC_H4

IP_H1 IP_H1 IP_H3 IP_H4
VNI1 VNI2 VNI1 VNI2

Vlan-100 Vlan-200 Vlan-100 Vlan-200
BGP for VxLAN
Advantages
• Minimizes network flooding through protocol-driven host MAC/IP route
distribution and ARP suppression on the local VTEPs.
• Provides optimal forwarding for east-west and north-south bound traffic with the
distributed any-cast function
• Provides VTEP peer discovery and authentication which mitigates the risk of
rouge VTEPs in the VXLAN overlay network.
VxLAN Overview
Distributed Anycast Gateway
Spine1 Spine2
L3 Underlay
Unicast / Multicast Routing
VTEP-1 VTEP-2 VTEP-3 VTEP-3
SVI SVI SVI SVI

GW IP GW IP GW IP GW IP
GW MAC GW MAC GW MAC GW MAC
Host-A Host-B
Host-C
VxLAN Overview
Distributed Anycast Gateway - Configuration
• All VTEPs has same IP address for an L2 VNI

• Anycast Gateway MAC is global to each VTEP for all VNI’s for all Tenants
• One virtual MAC / VTEP
• All VTEPs should have same virtual MAC address
fabric forwarding anycast-gateway-mac 0001.0001.0001

!
interface Vlan100
no shutdown
vrf context test-evpn-tenant
ip address 172.16.1.254/24
fabric forwarding mode anycast-gateway
VxLAN Overview
ARP Suppression
• Hosts send out G-ARP when they come

online
IP Multicast Core
• Local leaf node receives G-ARP, 2 +
creates local ARP cache and advertises VTEP-1 VxLAN EVPN VTEP-2
to other leaf by BGP as route type 2 3 V V
• Remote leaf node puts IP-MAC info into VRF
A
VRF
A
remote ARP cache and supresses
incoming ARP request for this IP 4 MAC_H1
IP_H1
1 MAC_H2
IP_H2
• If IP info not found in ARP suppression
VNI1 VNI1
cache table, VTEP floods the ARP Vlan-100 Vlan-100
request to other VTEPs
VxLAN Overview
Different Integrated Route/Bridge (IRB) Modes
Asymmetric IRB Symmetric IRB
• Uses different path from source to • Uses same path from source to
destination and back destination and back
• Required to configure the source • the ingress VTEP routes packets
VTEP with both the source and from source VNI to L3 VNI where
destination VNIs for both layer 2 and the destination MAC address in the
layer 3 forwarding inner header is rewritten to egress
VTEP’s router MAC address
VxLAN Overview
Different Integrated Route/Bridge (IRB) Modes
Asymmetric IRB
• Uses different path from source to
destination and back
IP Transport Network
• Required to configure the source SVI SVI SVI SVI
VTEP with both the source and 100 100 100 100
destination VNIs for both layer 2 and V1 V2

VTEP-1 VTEP-2
layer 3 forwarding
• Not a scalable method MAC_H1 MAC_H2
IP_H1 IP_H2
VNIA VNIB
VxLAN Overview Egress VTEP decapsulate
the packet and looks at
Symmetric IRB the inner packet header.
Ingress VTEP routes
Src-IP: VTEP-1 D.MAC address in the
Dst-IP: VTEP-2
packets from source VNI VNI: L3 VNI
inner packet header is its
to L3 VNI as DMAC is own Router MAC, it
S-MAC: Router-MAC-1
router MAC. D-MAC in D-MAC: Router-MAC-2 performs a Layer-3 routing
the inner header is
Src-IP: IP_H1 lookup. Layer-3 VNI in the
Dst-IP: IP_H2
rewritten to the Egress 1 2 VXLAN header provides
VTEP router MAC the VRF context to look
Encodes the Layer-3 up
VNI in the VXLAN IP Transport Network
header VNI L3 L3 VNI
1 VNI VNI A
V1 V2
VTEP-1 VTEP-2
Cisco follows
Symmetric IRB
MAC_H1 MAC_H2
IP_H1 IP_H2
VNIA VNIB
VxLAN Overview
VXLAN: Flood-&-Learn vs EVPN Control Plane
Flood-&-Learn EVPN Control Plane
Overlay Services L2+L3 L2+L3
Underlay Network IP network with ECMP IP network with ECMP
Encapsulation MAC in UDP MAC in UDP
Peer Discovery Data-driven flood-&-learn MP-BGP
Peer Authentication Not available MP-BGP
Host Route Learning Local hosts: Data-driven flood-&-learn Local Host: Data-driven
Remote hosts: Data-driven flood-&-learn Remote host: MP-BGP
Host Route Distribution No route distribution. MP-BGP
L2/L3 Unicast Forwarding Unicast encapsulation Unicast encapsulation
BUM Traffic forwarding Multicast replication Multicast replication

Unicast/Ingress replication Unicast/Ingress replication
VxLAN BGP EVPN
Configuration
VxLAN Configuration L2 VNI - 10000
VNI10000 - 239.1.1.1
Topology Spine Spine L3 VNI - 20000
10.10.10.10 20.20.20.20
Anycast RP – 11.11.11.11
L3 Underlay
Unicast / Multicast Routing
VTEP-3
192.168.100.100
Leaf1 (VTEP-1) Leaf2 (VTEP-2) Leaf3 (VTEP-3) Leaf4 (VTEP-3)

192.168.1.1 192.168.2.2 192.168.3.3 192.168.4.4
VPC
Host-A
Host-B
Host-C
Configuration
Feature Enablement
feature bgp For BGP Configuraiton

Feature pim
feature interface-vlan For PIM / Multicast Core
feature vn-segment-vlan-based
feature nv overlay SVI Configuration
!
For VxLAN Overlay
nv overlay evpn
! Map VLAN to VNI
Enables EVPN AFI

Not required to be
enabled on Spine
Leaf Node Configuration – L2 VNI
vlan 100 fabric forwarding anycast-gateway-mac 0001.0001.0001
vn-segment 10000 interface Vlan100
no shutdown
! Create L2 VNI vrf member EVPN-TENANT
ip address 100.1.1.254/24
evpn fabric forwarding mode anycast-gateway
vni 10000 l2 !
rd 10000:1 router bgp 100
route-target import 10000:1 router-id 192.168.1.1
route-target export 10000:1 address-family l2vpn evpn
! neighbor 192.168.10.10
interface nve1 remote-as 100
no shutdown update-source loopback0
source-interface loopback0 address-family l2vpn evpn
host-reachability protocol bgp send-community extended
member vni 10000 vrf EVPN-TENANT
mcast-group 239.1.1.1 address-family ipv4 unicast
suppress-arp advertise l2vpn evpn
Leaf Node Configuration – L3 VNI
vlan 200 interface nve1
vn-segment 20000 no shutdown
! source-interface loopback0
vrf context EVPN-TENANT host-reachability protocol bgp
vni 20000 member vni 20000 associate-vrf
rd 20000:1 !
address-family ipv4 unicast interface loopback200
route-target import 20000:1 vrf member EVPN-TENANT
route-target import 20000:1 evpn ip address 200.1.1.1/32
route-target export 20000:1 !
route-target export 20000:1 evpn router bgp 100
! vrf EVPN-TENANT
interface Vlan200 address-family ipv4 unicast
no shutdown network 200.1.1.1/32
vrf member EVPN-TENANT advertise l2vpn evpn
ip forward
!
Leaf Node with VPC Configuration
vpc domain 10
peer-switch
peer-keepalive destination 10.1.34.4 source 10.1.34.3
delay restore 60
peer-gateway
ipv6 nd synchronize VTEP IP. The secondary
ip arp synchronize IP is same on both Leaf3
! and Leaf 4 running VPC
interface loopback0
ip address 192.168.3.3/32
ip address 192.168.100.100/32 secondary
ip router ospf 100 area 0.0.0.0
ip pim sparse-mode
Spine Node Configuration
router bgp 100

interface loopback0
router-id 192.168.10.10
ip address 192.168.10.10/32
address-family l2vpn evpn
ip router ospf 100 area 0.0.0.0
neighbor 192.168.1.1
ip pim sparse-mode
remote-as 100
!
update-source loopback0
ip pim bsr-candidate loopback0
ip pim rp-candidate loopback0 group-list 239.1.1.0/24
send-community both
ip pim anycast-rp 192.168.10.10 192.168.1.1
route-reflector-client
ip pim anycast-rp 192.168.10.10 192.168.2.2
neighbor 192.168.2.2
ip pim anycast-rp 192.168.10.10 192.168.3.3
remote-as 100
ip pim anycast-rp 192.168.10.10 192.168.4.4
update-source loopback0
! Required on all nodes
send-community both
ip pim bsr listen forward
route-reflector-client
...
...
Leaf Node Configuration – L2 & L3 VNI for IPv6
vlan 100 fabric forwarding anycast-gateway-mac 0001.0001.0001
vn-segment 10000 interface Vlan100
vlan 200 no shutdown
vn-segment 20000 vrf member EVPN-TENANT
evpn ip address 100.1.1.254/24
vni 10000 l2 ipv6 address 2001::1/64
rd 10000:1 fabric forwarding mode anycast-gateway
route-target import 10000:1 !
route-target export 10000:1 interface nve1
! no shutdown
vrf context EVPN-TENANT source-interface loopback0
vni 20000 host-reachability protocol bgp
rd 20000:1 member vni 10000
address-family ipv4 unicast mcast-group 239.1.1.1
route-target both 20000:1 suppress-arp
route-target both 20000:1 evpn member vni 20000 associate-vrf
address-family ipv6 unicast !
route-target both 20000:1 router bgp 100
route-target both 20000:1 evpn vrf EVPN-TENANT
! address-family ipv4 unicast
interface Vlan200 advertise l2vpn evpn
no shutdown address-family ipv6 unicast
vrf member EVPN-TENANT advertise l2vpn evpn
ip forward !
ipv6 address use-link-local-only vpc domain 10
ipv6 nd synchronize
VxLAN EVPN Configuration
Host Learning and Peer Discovery
Host Learning Data Plane Control Plane
CORE
Multicast Flood and Learn EVPN-Multicast

Peer Learning: Data Plane Peer Learning: BGP-RnH
Vlan 100 Vlan 100

vn-segment 10000 vn-segment 10000
Interface nve1 Interface nve1
Member vni 10000 host-reachability protocol bgp
Mcast-group 239.1.1.1 member vni 10000
Mcast-group 239.1.1.1
Unicast Static Ingress-Replication EVPN Ingress-Replication

Peer Learning - CLI Peer Learning – BGP-IMET
Vlan 150 Vlan 150
vn-segment 15000 vn-segment 15000
Interface nve1 Interface nve1
member vni 15000 host-reachability protocol bgp
Ingress-replication protocol member vni 15000
static peer x.x.x.x ingress-replication protocol bgp
VxLAN BGP EVPN
Control-Plane
Verification
Nexus 9000 VxLAN Architecture
BGP
URIB, VxLAN
U6RIB Manager L2RIB
(PI)
HMM
MRIB
VxLAN
Manager Adj. Mgr
(PD)
MFDM L2FM ARP IPv6 ND
Troubleshooting VxLAN EVPN
VxLAN EVPN Prefix Types
Leaf1# show bgp l2vpn evpn
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10000:1 (L2VNI 10000)
*>i[2]:[0]:[0]:[48]:[8c60.4f19.51fc]:[0]:[0.0.0.0]/216
192.168.2.2 100 0 i
*>l[2]:[0]:[0]:[48]:[8c60.4f1b.e43c]:[0]:[0.0.0.0]/216
192.168.1.1 100 32768 i
*>i[2]:[0]:[0]:[48]:[8c60.4f19.51fc]:[32]:[100.1.1.2]/272
192.168.2.2 100 0 i
*>l[2]:[0]:[0]:[48]:[8c60.4f1b.e43c]:[32]:[100.1.1.1]/272
192.168.1.1 100 32768 i
Route Distinguisher: 20000:1

*>i[5]:[0]:[0]:[32]:[200.1.1.2]:[0.0.0.0]/224
192.168.2.2 100 0 i

*>i[2]:[0]:[0]:[48]:[8c60.4f19.51fc]:[32]:[100.1.1.2]/272
192.168.2.2 100 0 i
*>i[5]:[0]:[0]:[32]:[200.1.1.2]:[0.0.0.0]/224
192.168.2.2 100 0 i
EVPN Prefix Types
 BGP EVPN uses 5 different route types for IP prefixes

and advertisement Route TYPE - 8
 Type 1 - Ethernet Auto-Discovery (A-D) route
 Type 2 - MAC advertisement route  L2 VNI MAC/MAC-IP Length - 10
 Type 3 - Inclusive Multicast Route  EVPN IR, Peer Discovery
Route Type Specific
 Type 4 - Ethernet Segment Route
 Type 5 - IP Prefix Route  L3 VNI Route
 Route type 2 or MAC Advertisement route is for MAC

and ARP resolution advertisement, MAC or MAC-IP
 Route type 5 or IP Prefix route will be used for the
advertisement of prefixes, IP only
NVE Interface
Leaf1# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
Local Router MAC: f40f.1b6f.926f
Host Learning Mode: Control-Plane
Source-Interface: loopback0 (primary: 192.168.1.1, secondary: 0.0.0.0)
Leaf1# show interface nve1

nve1 is up If NVE Interface status is down, ensure
admin state is up, Hardware: NVE that a no shut is performed on the
MTU 9216 bytes
Encapsulation VXLAN
interface.
Auto-mdix is turned off
RX
ucast: 40 pkts, 5400 bytes - mcast: 1 pkts, 118 bytes
TX
ucast: 54 pkts, 6256 bytes - mcast: 9 pkts, 1026 bytes
Local MAC Routes Learning
L2FM
Mac Learnt on Vlan 100
Leaf1#show mac address-table vlan 100

(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+-------------
* 100 8c60.4f1b.e43c dynamic 0 F F Eth1/12
Leaf1# sh sys inter l2fm event-hist deb | in 8c60.4f93.5ffc

[104] l2fm_l2rib_add_delete_local_mac_routes(1095): To L2RIB: topo-id:
100, macaddr: 8c60.4f93.5ffc, nhifindx: 0x1a001600 peer_addr 0x1a001600
[104] l2fm_macdb_insert(6360): slot 0 fe 0 mac 8c60.4f93.5ffc vlan 100
flags 0x400107 hints 0 E8 NL lc : if_index 0x1a001600 old_if_index 0
L2FM installs the MAC in the L2RIB
L2FM L2RIB
Leaf1#show l2route evpn mac evi 100

Mac Address Prod Next Hop (s)
-------------- ------ ---------------
8c60.4f1b.e43c Local Eth1/12
Leaf1# show system internal l2rib event-history mac | in 8c60.4f93.5ffc

[06/01/16 22:31:55.201 UTC 5 9954] Received MAC ROUTE msg: addr: (100, 8c60.4f93.5ffc) vni: 0
admin_dist: 0 seq_num: 0 rt_flags: L soo: 0 dg_count: 0 res: 0 esi: (F) nh_count: 1
[06/01/16 22:31:55.202 UTC 7 9954] (100,8c60.4f93.5ffc):Mobility check for new rte from prod: 3
[06/01/16 22:31:55.202 UTC 8 9954] (100,8c60.4f93.5ffc):Current non-del-pending route local:no,
remote:no, linked mac-ip count:1
[06/01/16 22:31:55.202 UTC 9 9954] (100,8c60.4f93.5ffc,3):MAC route created with seq num:0, flags:L (),
soo:0, peerid:0
[06/01/16 22:31:55.205 UTC a 9954] (100,8c60.4f93.5ffc,3):Encoding MAC best route (ADD, client id 5)
[06/01/16 22:31:55.207 UTC 3 9954] (100,8c60.4f93.5ffc):Bound MAC-IP(100.1.1.1) to MAC, Total MAC-IP
linked: 1
L2 VNI, MAC
L2FM L2RIB BGP L2VPN
Leaf1#show bgp l2vpn evpn vni-id 10000

Network Next Hop Metric LocPrf Weight Path
*>l[2]:[0]:[0]:[48]:[8c60.4f93.5ffc]:[0]:[0.0.0.0]/216
192.168.1.1 100 32768 i
Leaf1# show bgp internal event-history events | in 8c60.4f93.5ffc

2016 Jun 1 22:31:55.205989 bgp 100 [16855]: [16888]: (default) RIB: [L2VPN EVPN
] add prefix 10000:1:[2]:[0]:[0]:[48]:[8c60.4f93.5ffc]:[0]:[0.0.0.0] (flags 0x1)
: OK, total 2
2016 Jun 1 22:31:55.205655 bgp 100 [16855]: [16888]: EVT: Received from L2RIB MAC
route: Add topo 10000 mac 8c60.4f93.5ffc soo 0 seq 0
. . .
Local MAC Address in BGP L2VPN
L2FM L2RIB BGP L2VPN
Leaf1#show bgp l2vpn evpn 8c60.4f93.5ffc

BGP routing table information for VRF default, address family L2VPN EVPN
BGP routing table entry for
[2]:[0]:[0]:[48]:[8c60.4f93.5ffc]:[0]:[0.0.0.0]/216,
version 8
Paths: (1 available, best #1)
Flags: (0x00010a) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
Path type: local, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path locally originated
192.168.1.1 (metric 0) from 0.0.0.0 (192.168.1.1)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 10000
Extcommunity: RT:10000:1
Remote L2 MAC Route Installation via BGP EVPN
Leaf2# show bgp l2vpn evpn 8c60.4f1b.e43c
BGP L2VPN BGP routing table information for VRF default, address family L2VPN
EVPN
BGP routing table entry for
[2]:[0]:[0]:[48]:[8c60.4f1b.e43c]:[0]:[0.0.0.0]/216, version 7
Flags: (0x00021a) on xmit-list, is in l2rib/evpn, is not in HW,
Path type: internal, path is valid, imported same remote RD, is
best path, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
192.168.1.1 (metric 5) from 192.168.10.10 (192.168.10.10)
Received label 10000
Extcommunity: RT:10000:1 ENCAP:8
Originator: 192.168.1.1 Cluster list: 192.168.10.10
EVPN BGP Route Type 2 Fields
 Ethernet Tag ID, MAC Address Length, MAC Address, IP Address Length, and IP Address fields are
considered to be part of the prefix in the NLRI.
 Ethernet Segment Identifier, MPLS Label1, and MPLS Label2 are treated as route attributes, not part of
the "route". Both the IP and MAC address lengths are in bits.
Leaf1#show bgp l2vpn evpn 8c60.4f93.5ffc Route Distinguisher – 8 byte

BGP routing table information for VRF default, address family L2VPN
EVPN Ethernet Segment ID – 10 byte
BGP routing table entry for Ethernet Tag ID – 4 byte
[2]:[0]:[0]:[48]:[8c60.4f93.5ffc]:[0]:[0.0.0.0]/216, version 8
Paths: (1 available, best #1) MAC Address Length – 1 byte
MAC Address – 6 byte
Path type: local, path is valid, is best path, no labeled nexthop IP Address Length – 1 byte
192.168.1.1 (metric 0) from 0.0.0.0 (192.168.1.1) IP Address – 0, 4, 16 byte
Received label 10000 MPLS Label 1 – 3 byte, L2VNI
Extcommunity: RT:10000:1
MPLS Label 2- 3 byte L3VNI
Remote L2 MAC Route Installation with BGP EPVN (Flow)
2a. Peer, VNI

Notification
BGP L2VPN
2b. Add (VNI, MAC-> Remote
VTEP IP)
VxLAN Mgr
VxLAN PD L2RIB
Lib.
5. Add (VNI,
3. Program data plane with
MAC -> Peer ID)
unicast encap/decap for VNI,
Allocate Peer ID
L2FM
UFDM
BGP to L2RIB
Leaf2# show bgp internal event-history events | in 8c60.4f1b.e43c

2016 Jun 2 02:53:14.844179 bgp 100 [9878]: [9890]: (default) IMP: bgp_tbl_ctx_import:
1812: [L2VPN EVPN] Importing
10000:1:[2]:[0]:[0]:[48]:[8c60.4f1b.e43c]:[0]:[0.0.0.0]/112 to RD 10000:1
2016 Jun 2 02:53:14.844167 bgp 100 [9878]: [9890]: (default) IMP: bgp_vrf_import:
2740: vrf default 10000:1:[2]:[0]:[0]:[48]:[8c60.4f1b.e43c]:[0]:[0.0.0.0]/112 result 1
2016 Jun 2 02:53:14.844130 bgp 100 [9878]: [9890]: (default) RIB: [L2VPN EVPN]: Send
to L2RIB 10000:1:[2]:[0]:[0]:[48]:[8c60.4f1b.e43c]:[0]:[0.0.0.0]/112 via 192.168.1.1
Add 1 EVPN MAC routes succeeded
Remote L2 MAC Route Installation via BGP EVPN
Leaf2# show nve internal bgp rnh database VxLAN Mgr

--------------------------------------------
Showing BGP RNH Database, size : 2 vni 0
VNI Peer-IP Peer-MAC Tunnel-ID Encap (A/S)

10000 192.168.1.1 0000.0000.0000 0x0 vxlan (1/0)
20000 192.168.1.1 f40f.1b6f.926f 0xc0a80101 vxlan (1/0)
BGP L2VPN
Leaf2# show l2route evpn mac evi 100

Mac Address Prod Next Hop (s) L2RIB
-------------- ------ ---------------
8c60.4f1b.e43c BGP 192.168.1.1
VxLAN Manager
Leaf2# show forwarding nve l3 peers
NVE cleanup transaction-id 0
tunnel_id Peer_id Peer_address Interface rmac origin state del count
--------------------------------------------------------------------------------------
0xc0a80101 1 192.168.1.1 nve1 f40f.1b6f.926f NVE merge-done no 1
Leaf2# show nve peers detail

Details of nve Peers:
----------------------------------------
Peer-Ip: 192.168.1.1
NVE Interface : nve1
VxLAN Mgr
Peer State : Up
Peer Uptime : 01:27:30 Hardware
Programs data plane with Router-Mac : f40f.1b6f.926f Programmed
unicast encap/decap for Peer First VNI : 20000
VNI, Allocate Peer ID
Time since Create : 01:27:30
Configured VNIs : 10000,20000
Provision State : add-complete
UFDM
Route-Update : Yes
Peer Flags : RmacL2Rib, TunnelPD, DisableLearn
Learnt CP VNIs : 10000,20000
Peer-ifindex-resp : Yes
L2FM Verification
Leaf2# show system internal l2fm debugs | in 8c60.4f1b.e43c

[104] l2fm_macdb_insert(6327): slot 32 fe 0 mac 8c60.4f1b.e43c vlan 100 flags 0x7
hints 0 E8 NL lc : if_index 0x49080001 old_if_index 0
[104] l2fm_l2rib_mac_update(21832): Add L2RIB remote mac 8c60.4f1b.e43c
[104] l2fm_process_l2rib_remote_route_update(405): Type: 2 Len: 152 Seq: 0, del: 0
(Prod: 5) Flags: Ctrl=3 Rt=0, mac 8c60.4f1b.e43c topo_id 100
Leaf2# show mac address-table vlan 100

---------+-----------------+--------+---------+------+----+-----------------
-
* 100 8c60.4f19.51fc dynamic 0 F F Eth1/13
* 100 8c60.4f1b.e43c dynamic 0 F F nve1(192.168.1.1)
Host IP and Host MAC local route
4. Route add: VRF test, Host-IP, L3VNI
HMM L2RIB
3. Adj. Notification: Host-IP, Host- 5. Route add: VRF test, Host-IP, L3 Table
MAC, SVI interface ID. Per prefix: L2 Table ID, Host-MAC
AM BGP
2. Create Adjacency: Host-IP, Host- L3 + L2 NLRI
MAC, Server facing SVI interface Prefix = Host-IP
MAC = Host-MAC
1b. Add an adjacency in AM, if the Label1: L3-VNI
ARP VLAN is enabled for L3 processing Label2: L2-VNI
VRF RT
1a. ARP Request/Reply BD RT
NH = VTEP IP
RMAC
ENCAP_VXLAN
Troubleshoot VxLAN EVPN
ARP from Host and ARP -> AM
Leaf1# show ip arp vrf EVPN-TENANT

IP ARP Table for context EVPN-TENANT
Total number of entries: 1
Address Age MAC Address Interface ARP
100.1.1.1 00:10:47 8c60.4f1b.e43c Vlan100
Leaf1# show forwarding vrf EVPN-TENANT adjacency

IPv4 adjacency information
AM
next-hop rewrite info interface
-------------- --------------- -------------
100.1.1.1 8c60.4f1b.e43c Vlan100
Troubleshoot VxLAN EVPN
AM
AM -> HMM -> L2RIB
Leaf1# show ip route vrf EVPN-TENANT

100.1.1.1/32, ubest/mbest: 1/0, attached
*via 100.1.1.1, Vlan100, [190/0], 02:41:57, hmm
100.1.1.254/32, ubest/mbest: 1/0, attached HMM
*via 100.1.1.254, Vlan100, [0/0], 02:59:46, local
Leaf1# show l2route evpn mac-ip evi 100

Mac Address Prod Host IP Next Hop (s)
-------------- ---- ------------------- -------------- L2RIB
-
8c60.4f1b.e43c HMM 100.1.1.1 N/A
L2RIB -> BGP
Leaf1# show bgp l2vpn evpn 100.1.1.1

BGP routing table entry for [2]:[0]:[0]:[48]:[8c60.4f1b.e43c]:[32]:[100.1.1.1]/272,
version 6
Path type: local, path is valid, is best path, no labeled nexthop
192.168.1.1 (metric 0) from 0.0.0.0 (192.168.1.1)
Received label 10000 20000
Extcommunity: RT:10000:1 RT:20000:1
Path-id 1 advertised to peers:

192.168.10.10
L2 + L3 Remote Route Installation
L3 + L2 NLRI BGP-EVPN
Prefix = Host-IP
MAC = Host-MAC EVPN Table
Label1: L3-VNI VTEP_IP,
Label2: L2-VNI RCAM, L3VNI,
VRF RT VRF Import Tunnel Info
L2VNI, VxLAN
BD RT BD Import
NH = VTEP IP RNH DB
RMAC VxLAN Manager
ENCAP_VXLAN Tunnel_ID
VRF BD
Peer-Id RMAC, VNI-
notification >Peer ID
VRF, Host-IP,
L3VNI, VTEP-IP
BD L2RIB
URIB
RMAC, VNI->Peer ID
VRF Program data plane
H1)-> Peer-ID
(L2-BD, MAC-
with unicast
VRF, Host-IP, encap/decap for
L3VNI, VTEP-IP L3VNI, RMAC,
Allocate Peer-ID
UFDM
FIB L2FM
L2RIB and URIB Information
Leaf2# show l2route evpn mac-ip evi 100

Mac Address Prod Host IP Next Hop (s)
-------------- ---- --------------------------- ---------------
8c60.4f1b.e43c BGP 100.1.1.1 192.168.1.1
Leaf2# show ip route vrf EVPN-TENANT 100.1.1.1

IP Route Table for VRF "EVPN-TENANT"
100.1.1.1/32, ubest/mbest: 1/0
*via 192.168.1.1%default, [200/0], 04:00:28, bgp-100, internal,
tag 100 (evpn) segid: 20000 tunnelid: 0xc0a80101 encap: VXLAN
Remote Host Prefix - EVPN
Leaf2# show bgp l2vpn evpn 100.1.1.1
version 5
Flags: (0x00021a) on xmit-list, is in l2rib/evpn, is not in HW, , is locked
Path type: internal, path is valid, imported same remote RD, is best path, no labeled
nexthop
192.168.1.1 (metric 5) from 192.168.10.10 (192.168.10.10)
Extcommunity: RT:10000:1 RT:20000:1 ENCAP:8 Router MAC:f40f.1b6f.926f
Path-id 1 not advertised to any peer

. . .
Remote Host Prefix – contd…
...
version 6
Flags: (0x00021a) on xmit-list, is in l2rib/evpn, is not in HW,
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from 10000:1:[2]:[0]:[0]:[48]:[8c60.4f1b.e43c]:[32]:[100.1.1.1]/144
(VNI 10000)
192.168.1.1 (metric 5) from 192.168.10.10 (192.168.10.10)
Extcommunity: RT:10000:1 RT:20000:1 ENCAP:8 Router MAC:f40f.1b6f.926f
Path-id 1 not advertised to any peer
NVE Internal Platform Detail
Leaf1# show nve internal platform interface nve 1 detail
Printing Interface ifindex 0x49000001 detail
|======|=========================|===============|===============|=====|=====|
|Intf |State |PriIP |SecIP |Vnis |Peers|
|======|=========================|===============|===============|=====|=====|
|nve1 |UP |192.168.1.1 |0.0.0.0 |2 |1 |
|======|=========================|===============|===============|=====|=====|
SW_BD/VNIs of interface nve1:

================================================
|======|======|=========================|======|====|======|========
|Sw BD |Vni |State |Intf |Type|Vrf-ID|Notified
|======|======|=========================|======|====|======|========
|100 |10000 |UP |nve1 |CP |0 |Yes
|200 |20000 |UP |nve1 |CP |3 |Yes
|======|======|=========================|======|====|======|========
Peers of interface nve1:
============================================
Peer_ip: 192.168.2.2
Peer-ID : 1
State : UP
Learning : Disabled
TunnelID : 0xc0a80202
MAC : 88f0.312a.f2c1
Table-ID : 0x1
Encap : 0x1
ARP Suppression Cache
Leaf1# show ip arp suppression-cache local

Ip Address Age Mac Address Vlan Physical-ifindex Flags
100.1.1.1 00:15:48 8c60.4f1b.e43c 100 Ethernet1/12 L
Leaf2# show ip arp suppression-cache remote

Ip Address Age Mac Address Vlan Physical-ifindex Flags
100.1.1.2 00:05:19 8c60.4f19.51fc 100 (null) R
Leaf1(config)# hardware access-list tcam region arp-ether 256
NVE Internal Information for Leaf Nodes with VPC Peers
Leaf3# show nve internal platform interface nve 1 detail
|======|=========================|===============|===============|=====|=====|
|======|=========================|===============|===============|=====|=====|
|nve1 |UP |192.168.3.3 |192.168.100.100|2 |2 |
|======|=========================|===============|===============|=====|=====|
================================================
|======|======|=========================|======|====|======|========
|======|======|=========================|======|====|======|========
|100 |10000 |UP |nve1 |CP |0 |Yes
|200 |20000 |UP |nve1 |CP |3 |Yes
|======|======|=========================|======|====|======|========

============================================
Peer_ip: 192.168.1.1
Peer-ID : 2
State : UP
Learning : Disabled
MAC : f40f.1b6f.926f
Table-ID : 0x1
Encap : 0x1
NVE Internal Information for Leaf Nodes on Non-VPC Peers
Leaf1# show nve internal platform interface nve1 detail
|======|=========================|===============|===============|=====|=====|
|======|=========================|===============|===============|=====|=====|
|nve1 |UP |192.168.1.1 |0.0.0.0 |2 |2 |
|======|=========================|===============|===============|=====|=====|
================================================
|======|======|=========================|======|====|======|========
|======|======|=========================|======|====|======|========
|100 |10000 |UP |nve1 |CP |0 |Yes
|200 |20000 |UP |nve1 |CP |3 |Yes
|======|======|=========================|======|====|======|========

============================================
Peer_ip: 192.168.100.100
Peer-ID : 2
State : UP
Learning : Disabled
MAC : 88f0.312b.9e4d
Table-ID : 0x1
Encap : 0x1
Silent Hosts
• A regular host initiates an ARP request to discover its Default Gateway
• ARP entry stored in cache, which times out after a certain time interval
• End host resends ARP request, to refresh the ARP entry => prevents from timing out
• If the TCP/IP stack for a host does not regularly send ARP packets to default
gateway or does not respond to ARP requests from default gateway, the ARP
entry times out and never reinstated again
• Such hosts are called Silent Hosts
• Rarely seen – Clustering applications with emulated virtual IP can result in silent
hosts
Solution for Silent Hosts
• On leafs, redistribute direct routes into VRF BGP.
• This generates type 5 EVPN for VXLAN subnet and border leaf will receive
VXLAN subnet routes from multiple leafs that has same subnet and put ECMP
routes in VRF RIB.
router bgp 100

vrf EVPN-TENANT
advertise l2vpn evpn
redistribute route-map SHOST-EVPN
• The Border Leaf receives subnet routes from multiple leafs that has SVI with
same subnet
• Border Leaf VRF RIB install ECMP routes to these leafs for host subnets
Solution for Silent Hosts contd…
• Those host subnet routes can be summarized and advertised to outside world
• When external network is trying to connect silent hosts, traffic comes to Border
Leaf and Border leaf has silent host subnet routes from multiple leafs and
choose one path from ECMP.
• Border leaf will forward the traffic to one of leaf that has silent host subnet SVI.
• The leaf that received the traffic will generate ARP as destination is attached
subnet and flood ARP in VxLAN.
• Silent host will receive it and reply back to the Leaf and traffic reaches silent
host.
• When silent host sent ARP reply, all Leafs will get silent host’s MAC+IP
information.
VxLAN Data Path
Verification
VxLAN L2 Unicast Hardware Verification
Encapsulation Path
• Step 1 – Verify NVE peers are UP

• Show nve peers
Leaf1# show nve peers
Inter Peer-IP State LearnType Uptime Router-Mac
---- --------- ------- -------- ----- ---------
nve1 192.168.2.2 Up CP 03:45:33 88f0.312a.f2c1
• Step 2 – Verify peer-id is allocated

• Show forwarding distribution peer-id
Leaf1# show forwarding distribution peer-id

UFDM Peer-id allocations:
App: VXLAN Vlan: 1 Id: 0x1c0a80202 Peer-id: 0x1
Encapsulation Path Contd…
• Verify MAC Address present in MAC Table
---------+-----------------+--------+---------+------+----+------------------
* 100 8c60.4f93.5ffc dynamic 0 F F Eth1/12
* 100 8c60.4f93.647c dynamic 0 F F nve1(192.168.2.2)
G 100 d8b1.9076.9053 static - F F sup-eth1(R)
• Verify MAC Address in Hardware Table VLAN = 28672 + VLAN-ID
Leaf1# show hardware mac address-table 1 vlan 100 address 8c60.4f93.647c

FE |PI| VLAN | MAC |Trunk| TGID |Mod|Port|Virtual|Static|Hit|Hit|CPU|Pending
| | | | | | | | Port| | SA| DA| |
---+--+------+---------------+-----+---------+---+----+-------+------+---+---+---+--------
0 1 28772 8c60.4f93.647c 0 0 0 0 31bc 0 1 1 0 0
Encapsulation Path Contd…
• Verify the MAC address in hardware using BCM Shell. Note the GPORT.
Leaf1# bcm-shell module 1 'l2 show' | grep 8c:60:4f:93:64:7c

mac=8c:60:4f:93:64:7c vlan=28772 GPORT=0x800031bc port=0x800031bc(vxlan) Hit
• Verify peer Next-Hop Info

Leaf1# bcm-shell module 1 'd chg ing_dvp_table 0x31bc'
ING_DVP_TABLE.ipipe0[12732]:
<VP_TYPE=3,NEXT_HOP_INDEX=0x11,NETWORK_PORT=1,ECMP_PTR=0x11,DVP_GROUP_PTR=0x11,>
• If ECMP=1 is set in the above output, that implies the route towards the peer is
an ECMP route
Verify SIP, DIP, DPORT
• Verify the DIP and get the Tunnel ID which has the SIP programmed
Leaf1# bcm-shell module 1 'd chg egr_dvp_attribute 0x31bc'
EGR_DVP_ATTRIBUTE.epipe0[12732]:
<VXLAN:TUNNEL_INDEX=1,VXLAN:DVP_IS_NETWORK_PORT=1,VXLAN:DIP=0xc0a80202,VXLAN:DE
LETE_VNTAG=1,VP_TYPE=2,>
• Verify the SIP using the Tunnel ID

Leaf1# bcm-shell module 1 'd chg egr_ip_tunnel 1'
EGR_IP_TUNNEL.epipe0[1]:
<TUNNEL_TYPE=0xb,TTL=0xff,SIP=0xc0a80101,L4_DEST_PORT=0x12b5,ENTRY_TYPE=1,DSCP_
SEL=1,>
0x12b5 = 4789
(VxLAN UDP port)
Verify VNI is created for VLAN and VNI is programmed in Hardware
• Verify the VLAN associated with the VNI
Leaf1# show nve vni 10000
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags

--------- -------- ----------------- ----- ---- ------------------ -----
nve1 10000 239.1.1.1 Up CP L2 [100] SA
• Verify VNI is created for VLAN 100

Leaf1# bcm-shell module 1 'd chg vfi 100'
VFI.ipipe0[100]:
<VP_1=0xffe,VP_0=0x1ffd,UUC_INDEX=0x1ffd,UMC_INDEX=0x1ffd,RSVD_VP_0=1,FLEX_CTR_P
OOL_NUMBER=4,FLEX_CTR_BASE_COUNTER_IDX=8,BC_INDEX=0x1ffd>
Verify VN ID programmed in Hardware for VLAN
• VNI 10000 = 0x2710
• VLAN 100 = 0x64
Leaf1# bcm-shell module 1 'd chg mpls_entry' | grep 0x2710

MPLS_ENTRY.ipipe0[8680]:
<VXLAN_VN_ID:VN_ID=0x2710,VXLAN_VN_ID:VFI=0x64,VXLAN_VN_ID:KEY=0x00000000000271
09,VXLAN_VN_ID:HASH_LSB=0x710,VXLAN_VN_ID:DATA=0x64,VALID=1,KEY_TYPE=9,>
Decapsulation Path – Network to Access Direction
• Verify the VTEP Gateway is enabled and correct Loopback IP is used
Leaf1# show nve interface

Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
Local Router MAC: d8b1.9076.9053
Host Learning Mode: Control-Plane
Source-Interface: loopback0 (primary: 192.168.1.1, secondary: 0.0.0.0)
Leaf1# show nve vxlan-params

VxLAN Dest. UDP Port: 4789
Verify Tunnel Termination is programmed in hardware
• Verify VxLAN_XLATE Entry is created with local NVE loopback address
Leaf1# bcm-shell module 1 'd chg vlan_xlate' | grep DIP
VLAN_XLATE.ipipe0[3240]:
<VXLAN_DIP:KEY=0x6054008092,VXLAN_DIP:IGNORE_UDP_CHECKSUM=1,VXLAN_DIP:HASH_LSB=0
x101,VXLAN_DIP:DIP=0xc0a80101,VXLAN_DIP:DATA=0x00000000000000000000000000400000,
VALID=1,KEY_TYPE=0x12,>
• Verify the MPLS entry is programmed for the peer SIP in hardware
Leaf1# show forwarding distribution peer-id
UFDM Peer-id allocations:
App: VXLAN Vlan: 1 Id: 0x1c0a80202 Peer-id: 0x1
Leaf1# bcm-shell module 1 'd chg mpls_entry' | grep 0xc0a80202
<VXLAN_SIP:SVP=0x31bc,VXLAN_SIP:SIP=0xc0a80202,VXLAN_SIP:KEY=0x0c0a802028,VXLAN_
SIP:HASH_LSB=0x202,VXLAN_SIP:DATA=0x31bc,VALID=1,KEY_TYPE=8,EVEN_PARITY=1,>
Verify Tunnel Termination is programmed in hardware
• Verify MPLS entry is programmed for the VNI to VLAN conversion in hardware
• VLAN 100 = 0x64
• VNI 10000 = 0x2710
Leaf1# bcm-shell mod 1 'd chg mpls_entry' | grep 0x2710

<VXLAN_VN_ID:VN_ID=0x2710,VXLAN_VN_ID:VFI=0x64,VXLAN_VN_ID:KEY=0x000000000002710
9,VXLAN_VN_ID:HASH_LSB=0x710,VXLAN_VN_ID:DATA=0x64,VALID=1,KEY_TYPE=9,>
Packet Path after Decap
• After decap, if inner DMAC is learned in a VxLAN VLAN, then packet will be sent
out using the nexthop information of the mac.
---------+-----------------+--------+---------+------+----+------------------
•
* 100 8c60.4f93.5ffc dynamic 0 F F Eth1/12
• Verify the MAC address in hardware

Leaf1# show hardware mac address-table 1 vlan 100 address 8c60.4f93.5ffc
FE |PI| VLAN | MAC |Trunk| TGID |Mod|Port|Virtual|Static|Hit|Hit|CPU|Pending
| | | | | | | | Port| | SA| DA| |
---+--+------+---------------+-----+---------+---+----+-------+------+---+---+---+--------
0 1 28772 8c60.4f93.5ffc 0 0 0 0 1641 0 0 1 0 0
28672 + VLAN-ID = VLAN

Verify hardware is programmed pointing to correct access interface
• Verify Ingress Port for Virtual Port
Leaf1# bcm-shell module 1 'd chg ing_dvp_table 0x1641'
ING_DVP_TABLE.ipipe0[5697]: <NEXT_HOP_INDEX=0xd,ECMP_PTR=0xd,DVP_GROUP_PTR=0xd,>
• Verify Next Hop information for the Next-Hop Index
Leaf1# bcm-shell module 1 'd chg ing_l3_next_hop 0xd'

ING_L3_NEXT_HOP.ipipe0[13]:
<VLAN_ID=0xffe,TGID=0x98,PORT_NUM=0x18,MTU_SIZE=0x1ffe,MODULE_ID=1,L3_OIF=0x1ffe
,ENTRY_INFO_UPPER=1,DVP_RES_INFO=0x7e,>
Port Number mapping to Physical Port
• Verify the Mapping of Physical port to Port Number
Leaf1# bcm-shell module 1 'phy info'
Phy mapping dump:
port id0 id1 addr iaddr name timeout
hg0( 1) 600d 8770 1b1 1b1 TSC-A2/31/4 250000
hg1( 2) 600d 8770 81 81 TSC-A2/00/4 250000
. . .
xe11( 24) 600d 8770 b8 b8 TSC-A2/10/3 250000
Leaf1# show interface hardware-mappings

--------------------------------------------------------------------
Name Ifindex Smod Unit HPort FPort NPort VPort
--------------------------------------------------------------------
Eth1/12 1a001600 1 0 24 255 11 -1
VxLAN OAM
VxLAN OAM
NGOAM
• New feature introduced in 7.0(3)I6(1) – NGOAM
• Need a feature for Path verification and Path
tracking with Telemetry data
• Similar to Fabric Path OAM
Enable NGOAM
• Provides 3 features Feature
• VxLAN Ping feature ngoam
• VxLAN Traceroute
Create Profile
ngoam profile 1
• VxLAN PathTrace oam-channel 2
! Install ACL
ngoam install acl
VxLAN OAM
Usability
• Helps diagnose underlay / overlay reachability of VMs
• Covers exact path as Data Packet
• Path verification for all ECMP paths in Overlay
• Path tracking – Exact path host traffic takes in overlay and underlay network
• Layer 2 – Traceroute / Ping to VM host from Leaf
• Layer 3 – Traceroute / Ping to Vm host from Leaf
• Flexible OAM channel supporting multiple drafts
• Tissa draft – nvo3
VxLAN OAM
VxLAN OAM Message
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
. Circuit Header . (variable)
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Transport Header + Technology dependent
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ VXLAN Header + 8 bytes
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
. OAM Message Channel . Variable
. .
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ Link Trailer + Variable
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
VxLAN OAM
VxLAN PING
• VxLAN Ping checks connectivity to the destination, where the destination can be
VM’s IP address or routed loopback addresses on the leaf switch
• Since there are multiple paths, only one path is followed based on the flow
parameters
• Ping for both VM / Host MAC and IP
• Default ping support – Ping based on just destination address and VNI segment
• Allow users to specify flow parameters such as UDP port, destination and
source address
• This helps VxLAN ping follow the specific path the unicast ping will take to reach the
destination
VxLAN OAM
PING NVE MAC
Leaf1# ping nve mac 0050.56b3.bcef 200 port-channel 101
profile 1 vni 20000 verbose
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'D' - Destination Unreachable, 'X' - unknown return code,
'm' - malformed request(parameter problem)
,'c' - Corrupted Data/Test, '#' - Duplicate response
Sender handle: 21
! sport 51932 size 39,Reply from 192.168.100.100,time = 5 ms
VxLAN OAM
VxLAN Traceroute
• VxLAN Traceroute – Used to trace the path a packet takes between source and
destination
• Only one path is traced based on the given flow parameters
• Trace will show uni-directional path the packet takes to the destination, but the return
path may be different
• Should be able to trace VTEPs, access switch and end-host. For the default
mode, the user should be able to trace the tunnel endpoint IP address and the
segment ID
• Actual path taken by a packet is dependent on all the L2/L3/L4 header fields and
network topology at the time the packet is sent
• Users can specify the flow parameters
VxLAN OAM
VxLAN PathTrace
• Similar to Traceroute, but uses Nvo3 channel
• Carries additional ingress / egress and load information of the path
• Provides additional information if the device supports nvo3 channel else its same as
traceroute
• Actual path taken by a packet is dependent on all the L2/L3/L4 header fields and
network topology at the time the packet is sent
• Users can specify the flow parameters
Leaf1# pathtrace nve mac 0050.56b3.bcef 200 port-channel 101 vni 20000
<snip>
Path trace Request to peer ip 192.168.100.100 source ip 192.168.99.99
Sender handle: 35Hop Code ReplyIP IngressI/f EgressI/f State
====================================================
1 !Reply from 10.101.1.10, Eth2/1 Eth1/17 UP / UP
2 !Reply from 192.168.100.100, Eth1/17 Unknown UP / DOWN
Case Studies
Spine1 Spine2
Anycast RP
Case Study 1
Leaf1 - Lo0 - 192.168.1.1/32 11.11.11.11/32
Leaf2 - Lo0 - 192.168.2.2/32
Leaf3 - Lo0 - 192.168.3.3/32

Leaf4 - Lo0 - 192.168.4.4/32
Spine1 - Lo0 - 10.10.10.10/32

Spine2 - Lo0 - 20.20.20.20/32
192.168.99.99 192.168.100.100
L2 VNI - 10000, 20000 BL2

Leaf1 Leaf2 Leaf3 Leaf4 BL1
L3 VNI - 30000
VNI 10000 - 239.1.1.1

VNI 20000 - 239.2.2.2
CRS1 CRS2
External Router 10.111.121.1

H1 H2 H2 H5 H5
V-100 V-100 V-200 V-100 V-200
Case Study 1
Description
• If the traffic from host 10.111.121.1 takes the path via CRS2, the host is unable
to reach 100.1.1.1
• If the traffic from host 10.111.121.1 takes the path via CRS1, then the host is
able to reach 100.1.1.1
• Leaf1 can see BL1 in NVE peer list but not BL2 Question: Why
No entry for
Leaf1# sh nve peers BL2
Interface Peer-IP State LearnType Uptime Router-Mac (192.168.6.6)
--------- --------------- ----- --------- -------- -----------------
nve1 192.168.5.5 Up CP 00:00:56 ecbd.1d0b.bb53
nve1 192.168.100.100 Up CP 23:51:43 003a.7d61.7681
• All BL nodes can see all Leaf nodes in NVE peer list
Case Study 2
Spine1 Spine2
Ingress Replication
with • Communication not happening
BGP EVPN
between host H1 and host H5
L2 VNI - 10000
L3 VNI - 11000 • Performing tcpdump on host H1, we
192.168.99.99 192.168.100.100
L2 VNI - 20000
L3 VNI - 21000
can see packet from H5 is able to
reach H1
Leaf1 Leaf2 Leaf3 Leaf4 • No packet from H1 to H5
• Customer is performing Inter-VRF
Route Leaking
• Routes seen in both BGP table and
VRF Routing table
• All other components on leaf are
H1
V-100
H2 H2
V-100 V-200
H5 H5
V-100 V-200
having relevant information
Case Study 2
Steps to Troubleshoot
• Verify if the packet from H1 is reaching Leaf1 / Leaf2
• If there is a firewall or any other device before Leaf1 / Leaf2, verify if the packet is
reaching to that point
• On Firewall perform packet capture
• On other devices perform other packet capture techniques such as SPAN / ERSPAN, EPC, etc.
• To ensure the packet is reaching Leaf1 or Leaf2, perform Ethanalyzer or ELAM for
regular IP packet
• Verify if the VxLAN packet is reaching Leaf 3 or Leaf4
• Perform ELAM for VxLAN encapsulated packet
• Verify if the packet is able to reach server below Leaf3
• Other packet capture techniques such as SPAN, tcpdump on server, etc.
Case Study 2
ELAM
• Embedded Logic Analyzer module (ELAM) – tool used to capture a packet
processed by a Cisco ASIC
• Depending on the N9k platform,
• ELAM on NS ASIC
• ELAM on TAHOE ASIC
• Useful in scenario’s where packet forwarding is impacted

• Can perform capture for raw packet from the host and even VxLAN
encapsulated packet towards the VxLAN Core
• Useful for Cisco TAC and Cisco Engineering for understanding the cause of
packet loss or impacted forwarding
ELAM on N9k Platform
Northstar (NS) ASIC
• NS datapath is divided into ingress and egress pipelines
• 2 ELAM’s present in each pipeline (Input & Output)
• Packets can be captured based on either input select lines or output select lines but not
both
• Input Select Lines
• 3  Outerl2-outerl3-outerl4
• 4  Innerl2-innerl3-inner l4
• 5  Outerl2-innerl2
• Output Select Lines

• 0  Pktrw
• 5  Sideband
NS ELAM Steps
• Attach module X
• Debug platform internal ns elam asic [0 | 1]
• Trigger [init | reset] [ingress | egress] in-select [3 – 7]
out-select [0, 5]
• Ingress – For capturing packets from front panel to Fabric
• Egress – For capturing packets from Fabric to Front Panel
• Set outer [ipv4 | l2 | l4] . . . .

• Start
• Status {Can be Armed / Triggered}
• report
Verify Port-Asic, Slice and Src-Id
N9K# show hardware internal tah interface e1/49
<snip>
IfIndex: 436232192
DstIndex: 5952
IfType: 26
Asic: 0
AsicPort: 56
SrcId: 48 <<
Slice: 1
PortOnSlice: 24
TAHOE ELAM Steps
• Attach module X
• Debug platform internal tah elam asic [0 | 1]
• Trigger [init | reset] asic [num] slice [num] lu-a2d [0 |
1] in-select [3-7] out-select [0-5] use-src-id [src-id]
• Lu-a2d 0 – used for reverse ELAM, where trigger is based on result
• Lu-a2d 1 – used for ELAM where trigger is based on packet attributes
• Set outer [ipv4 | l2 | l4] . . . .

• Start
• Status {Can be Armed / Triggered}
• Report [detail]
TAHOE ELAM
Spine
att mod 1
debug platform internal tah elam asic 0
trigger init asic 0 slice 0 lu-a2d 1 in-select 7 out-select 0 use-src-id 52
reset VxLAN Encapsulated Packet
set inner ipv4 src_ip 100.1.1.1 dst_ip 100.1.1.5
start
report
Leaf1
att mod 1
debug platform internal tah elam asic 0
trigger init asic 0 slice 0 lu 1 in-select 6 out-select 0
reset Host IP Packet
set outer ipv4 src_ip 100.1.1.1 dst_ip 100.1.1.5
start
report
Partial Output
• Dot1Q Header
module-1(TAH-elam-insel6)# report | grep pr_lu_vec_l2v.qtag0
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_vld: 0x1 << dot1q yes? 0x1
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_cos: 0x0
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_de: 0x0
GBL_C++: [MSG] pr_lu_vec_l2v.qtag0_vlan: 0x64 << VL 100
• VLAN
module-1(TAH-elam-insel6)# report | grep -1
fpx_lookup_vec.lkup.macsakey.key.fid
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.vld: 0x1
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.fid_type: 0x0
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.fid_vld: 0x0
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.fid: 0x64 << dec 0xa = VL 100
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.mac: 0xFEC80E2715
Partial Output
• Src & Dst IP
module-1(TAH-elam-insel6)# report | grep vec_l3v.ip.*a
GBL_C++: [MSG] pr_lu_vec_l3v.ip.da: 0x0000000000000000064010101 << 100.1.1.1
GBL_C++: [MSG] pr_lu_vec_l3v.ip.sa: 0x0000000000000000064010105 << 100.1.1.5
• Src MAC
module-1(TAH-elam-insel6)# report | grep -i
fpx_lookup_vec.lkup.macsakey.key.mac
GBL_C++: [MSG] fpx_lookup_vec.lkup.macsakey.key.mac: 0xFEC80E2715 <<
00fe.c80e.2715
Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you

BRKDCN 3040

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN 3040

Uploaded by

Copyright:

Available Formats

Troubleshooting

VxLAN BGP EVPN

Vinit Jain - CCIE# 22854

Cisco Spark spaces will be cs.co/ciscolivebot#BRKDCN-3040

• VxLan Overview • Troubleshooting VxLAN EVPN

VTEP – VXLAN Tunnel End-Point

Outer Outer Inner

Flags Reserved Instance ID Reserved

8 Bytes 24 Bytes 24 Bytes 8 Bytes

• VXLAN to VLAN Bridging V

Ingress VXLAN packet Egress VXLAN packet is ROUTED

• VXLAN-to-VLAN Routing Ingress VXLAN packet

Segment IDs are utilized for providing isolation at

802.1Q tagged frames received at the Leaf nodes

The VLAN to Segment mapping is performed on a

Thus, Segment IDs are globally significant and

Outer S-IP: 192.168.1.1 MAC-A 10 192.168.1.1

• Leverages the decade-long MP-BGP VPN technology to support scalable

• EVPN address family carries both Layer 2 and Layer 3 reachability

SVIX SVIA SVI

Layer-3 VNI X’ Layer-2 VNI A’ Layer-2 VNI B’

VLAN X VLAN A VLAN B

• 1 Layer-3 VNI per • 1 Layer-2 VNI per Layer-2 segment

Mac_H1: VNI1 Mac_H3: VNI1

MAC_H1 MAC_H1 MAC_H3 MAC_H4

VNI1 VNI2 VNI1 VNI2

VTEP-1 VTEP-2 VTEP-3 VTEP-3

SVI SVI SVI SVI

• All VTEPs has same IP address for an L2 VNI

fabric forwarding anycast-gateway-mac 0001.0001.0001

• Hosts send out G-ARP when they come

destination VNIs for both layer 2 and V1 V2

Flood-&-Learn EVPN Control Plane

Overlay Services L2+L3 L2+L3

Underlay Network IP network with ECMP IP network with ECMP

Encapsulation MAC in UDP MAC in UDP

Peer Discovery Data-driven flood-&-learn MP-BGP

Peer Authentication Not available MP-BGP

Host Route Distribution No route distribution. MP-BGP

L2/L3 Unicast Forwarding Unicast encapsulation Unicast encapsulation

BUM Traffic forwarding Multicast replication Multicast replication

Leaf1 (VTEP-1) Leaf2 (VTEP-2) Leaf3 (VTEP-3) Leaf4 (VTEP-3)

feature bgp For BGP Configuraiton

Enables EVPN AFI

router bgp 100

Multicast Flood and Learn EVPN-Multicast

Vlan 100 Vlan 100

Unicast Static Ingress-Replication EVPN Ingress-Replication

MFDM L2FM ARP IPv6 ND

Route Distinguisher: 20000:1

Route Distinguisher: 20000:1 (L3VNI 20000)

 BGP EVPN uses 5 different route types for IP prefixes

 Route type 2 or MAC Advertisement route is for MAC

Leaf1# show interface nve1

Leaf1#show mac address-table vlan 100

Leaf1# sh sys inter l2fm event-hist deb | in 8c60.4f93.5ffc

Leaf1#show l2route evpn mac evi 100

Leaf1# show system internal l2rib event-history mac | in 8c60.4f93.5ffc

Leaf1#show bgp l2vpn evpn vni-id 10000

Leaf1# show bgp internal event-history events | in 8c60.4f93.5ffc

Leaf1#show bgp l2vpn evpn 8c60.4f93.5ffc

Leaf1#show bgp l2vpn evpn 8c60.4f93.5ffc Route Distinguisher – 8 byte

2a. Peer, VNI