MPLS VPN Troubleshooting

Troubleshooting MPLS VPN Networks
MPLS VPN Trouble
2004 Cisco Systems, Inc. All rights reserved.
Agenda
Control Plane Troubleshooting
Forwarding Plane Troubleshooting
Conclusion
MPLS VPN Trouble
Agenda
Control Plane
Control Plane Troubleshooting Tips
Real-life Examples
Summary of Helpful Cisco IOS Commands
Forwarding Plane
Dissecting LFIB
Load sharing in MPLS VPN Networks
Forwarding Plane Troubleshooting Tips
Real-life Examples
Conclusion
MPLS VPN Trouble
MPLS VPN - Troubleshooting Tips

Symptom
Tip
Cisco IOS
Command
VPNv4 Prefix Is Not Received at the

Remote (Receiving) PE
Make Sure that export RT <X> at

the Advertising PE Matches with
import RT <X> at the Receiving
PE
Sh ip vrf detail <vrf> | inc

Export|import|RT

Validate the Match/Set Clause

within the Export-Map or ImportMap (if Any)
sh ip vrf de <vrf> | inc

route-map;

If BGP Is Not the Chosen PE-CE

Protocol, then Validate BGP->IGP
Redistribution
sh run | b router <igp>

Check whether the Remote PE Is

Configured as the rr-client within
VPNv4 af at the Route-Reflectors
Sh run | b address-family
vpnv4
4
MPLS VPN Trouble
sh route-map <map>
MPLS VPN - Troubleshooting Tips (Cont.)

Cisco IOS
Command
Symptom
Tip
VPNv4 Prefix Is Not Received

at the Remote (Receiving) PE
Make Sure that the RouteReflectors and PEs Are

Configured to Send ExtCommunity towards the
iBGP Peers within the
VPNv4 af
sh run | b addressfamily vpnv4
VPNv4 Traffic Is Not Getting

Forwarded End-to-End
Check the Label

Information in BGP and
LFIB at the Advertising PE
Router
sh ip bgp vpn vrf <vrf>

label | inc <prefix>
Check the Label

Information in BGP and
FIB at the Receiving PE
Router

label | inc <prefix>
6
VPNv4 Traffic Is Not Getting
Forwarded End-to-End
MPLS VPN Trouble
sh mpls for vrf <vrf> |

inc <prefix>
sh ip cef vrf <vrf>

<prefix>
MPLS VPN Label Stack
Outer (or IGP) label in the label stack provides a

LSP from ingress PE to egress PE via MPLS cloud
Inner (or BGP) label refers to the VPNv4 prefix at

the egress PE
tag rewrite with Se2/0, point2point, tags imposed: {2003 20}
MPLS VPN Trouble
Agenda
Control Plane
Real-life Examples
Forwarding Plane
Dissecting LFIB
Real-life Examples
Conclusion
MPLS VPN Trouble
MPLS VPN Real Life Examples

Lets do some MPLS VPN trouble(shooting)
MPLS VPN Trouble
MPLS VPN Ctrl PlaneTrouble #1

#1: VPN prefix doesnt have any label in the LFIB on
the local PE
PE1
Ser2/0
200.1.61.4/30
CE1
Loop0:10.13.1.61/32
PE1#sh mpls forwarding vrf v1 | i 200.1.61.4

PE1#
PE1#sh ip bgp vpn vrf v1 label | i 200.1.61.4
AS#1
PE1#
PE1#sh ip bgp vpn vrf v1 200.1.61.4
MPLS Backbone
%Network not in the table
PE1#
TIP: Label allocation is done by BGP. So make sure the prefix is in the BGP
VRF table. Hintredistribute connected
MPLS VPN Trouble
MPLS VPN Ctrl PlaneTrouble #1 (Cont.)

PE1(conf)#router bgp 1
PE1(conf-router)#address-family ipv4 vrf v1
PE1(conf-router-af)#redistribute connected
PE1(conf-router-af)#end
PE1
Ser2/0
200.1.61.4/30
CE1
MPLS VPN Trouble
PE1#sh ip bgp vpn vrf v1 label | i 200.1.61.4

200.1.61.4/30
0.0.0.0
30/nolabel
PE1#
AS#1PE1#sh mpls forwarding vrf v1 | i 200.1.61.4
30
Aggregate
200.1.61.4/30[V] 0
MPLS Backbone
PE1#
Loop0:10.13.1.61/32
As soon as BGP gets the VPN prefix, it allocates the local label, and
installs the prefix+label in both BGP and LFIB
10

#2: LFIB doesnt have any label for the VPNv4 prefix
at the local PE, though BGP now does.
TIP: clear ip route vrf <vrf> <prefix>
If the above doesnt fix, then (soft) reset the BGP session
MPLS VPN Trouble
11

#3: Remote PE (PE2) doesnt get the VPNv4 prefix from
PE1
!
RR1
PE1
AS#1
MPLS Backbone
Ser2/0
200.1.61.4/30
ip vrf v1
rd 1:1
route-target import 1:1
PE2
Loop0:10.13.1.62/32
CE-2
Loop0:10.13.1.61/32

% Network not in the table
PE2#
PE2#sh ip vrf de v1 | beg Import
No Import VPN route-target communities
No import route-map
No export route-map
PE2#
CE1
TIP: Validate route-target import config at PE2. If not present, then

configure it; Check for import-map as well
MPLS VPN Trouble
12

#4: Remote PE (PE2) still doesnt get the VPNv4 prefix
from PE1
RR1
PE1
AS#1
MPLS Backbone
Ser2/0
200.1.61.4/30
!
ip vrf v1
rd 1:1
route-target import 1:1
PE2
Loop0:10.13.1.62/32
CE-2
Loop0:10.13.1.61/32
CE1
MPLS VPN Trouble

% Network not in the table
PE2#
We already fixed PE2; so lets go to PE1
Validate Route-target export in the VRF at the PE1
13

PE1(conf)#ip vrf v1
PE1#sh ip bgp vpnv4 vpn vrf v1 200.1.61.4
PE1(conf-vrf)#route-target export 1:1 BGP routing table entry for 1:1:200.1.61.4/30, version 10
Paths: (2 available, best #2, table v1)
PE1(conf-vrf)#end
PE1
Ooops..RT Is Missing
Ser2/0
Advertised to non peer-group peers:

10.13.1.21 200.1.61.6
Local
0.0.0.0 from 0.0.0.0 (10.13.1.61)
Origin incomplete,
metric 0, localpref 100, weight
RR1
32768, valid, sourced, best
PE2
PE1#
AS#1
200.1.61.4/30
MPLS Backbone
Loop0:10.13.1.62/32
CE-2
Loop0:10.13.1.61/32
CE1
TIP: Configure Route-target export in the VRF on the local PE i.e. PE1
Lets make sure that RT is getting tagged to the VPNv4 prefix
MPLS VPN Trouble
14

PE1#sh ip bgp vpnv4
RR1 vpn vrf v1 200.1.61.4
BGP routing table entry for 1:1:200.1.61.4/30,
version 10
PE1
PE2
AS#1
Ser2/0
10.13.1.21
200.1.61.6
MPLS
Backbone
200.1.61.4/30
Loop0:10.13.1.61/32 Local
0.0.0.0 from 0.0.0.0 (10.13.1.61)
Origin incomplete, metric 0, localpref 100, weight
CE1
32768, valid, sourced, best
Extended Community: RT:1:1
RT is getting tagged
PE1#
Extra-TIP
If export or import map are also configured, then check the RT in set
clause, along with the match clause
MPLS VPN Trouble
15
MPLS VPN Ctrl PlaneTrouble#5

#5: Remote PE (PE2) STILL doesnt get the
VPNv4 prefix from PE1
RR1#sh ip bgp vpnv4 rd 1:1 200.1.61.4
BGP routing table entry for 1:1:200.1.61.4/30, version 14
Paths: (1 available, best
PE1
PE2#1, no table)
AS#1 10.13.1.62
CE-2
30
Local, (Received from a RR-client)
MPLS Backbone
.4/
1
Loop0:10.13.1.62/32
6
.
10.13.1.61 (metric 75) from 10.13.1.61 (10.13.1.61)
0.1
Loop0:10.13.1.61/32
20
Origin incomplete, metric 0, localpref 100, valid, internal, best
RR1#
RR1
CE1
Looks Good on RR1
RR1 is indeed receiving the prefix from PE1

Make sure that RR is configured with neighbor <PE2> send-community
extended under vpnv4 address-family
MPLS VPN Trouble
16

Ooops. PE2 i.e
10.13.1.62 Is Missing
RR1(conf)#router bgp 1
RR1(conf-router)#address-family vpnv4
RR1(conf-router-af)#neighbor 10.13.1.62
send-community extended
RR1(conf-router-af)#neighbor 10.13.1.62
route-reflector-client
PE1
RR1(conf-router-af)#end
Ser2/0
200.1.61.4/30
RR1#sh run | inc send-community ext

neighbor 10.13.1.61 send-community extended
PE1#
RR1#sh run | inc send-community ext
PE1#
RR1
AS#1
MPLS Backbone
PE2
Loop0:10.13.1.62/32
CE-2
Loop0:10.13.1.61/32
CE1
MPLS VPN Trouble
TIP: All the MP-BGP peers must be configured with

send-community extended|both
Also make sure that PE1 and PE2 are configured as

route-reflector-client under vpnv4 af at the RR1
17
MPLS VPN Control PlaneTrouble #6

#6: Remote PE (PE2) STILL doesnt get
the VPNv4 prefix from PE1
PE2#sh ip vrf detail v1 | i Import

Import route-map: raj-import
PE2#
PE2#sh route-map raj-import
RR1
route-map raj-import, permit,
PE1 sequence 10
PE2
Match clauses:
AS#1
extcommunity (extcommunity-list
filter):1
Ser2/0
Set clauses:
MPLS Backbone Loop0:10.13.1.62/32
Policy 200.1.61.4/30
routing matches: 0 packets, 0 bytes
Loop0:10.13.1.61/32
PE2#
PE2#sh
ip extcommunity-list 1
CE1
Extended community standard listPE2#sh
1
ip bgp vpn vrf v1 200.1.61.4
deny RT:1:1
% Network not in theOhtable
no.who did that
deny RT:2:2
PE2#
&^%@#%@^%
PE2#
CE-2
Thats ok. Lets

Remove RT 1:1
from the Filter.
Hmm we have already verified PE1 and RR1; something must be missing
on PE2 then
Lets check for any import-map at PE2 again
MPLS VPN Trouble
18
MPLS VPN Control PlaneTrouble #6 (Cont.)

PE2(conf)#no ip extcommunity-list 1 deny rt 1:1
RR1
PE2(conf)#end
PE1
Ser2/0
200.1.61.4/30
AS#1
MPLS Backbone
PE2
Loop0:10.13.1.62/32
CE-2
Loop0:10.13.1.61/32
CE1
PE#clear ip bgp * vpnv4 unicast in

PE2#sh ip bgp vpnv4 vrf v1 200.1.61.4
BGP routing table entry for 1:1:200.1.61.4/30, version 180
200.1.62.6
Local
10.13.1.61 (metric 75) from 10.13.1.21 (10.13.1.21)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Originator: 10.13.1.61, Cluster list: 10.13.1.21
PE2#
TIP: If import-map is configured within the VRF, then import route-target

<rt> must be configured within the VRF for the relevant RT
MPLS VPN Trouble
19

#7: Label mismatch between BGP and FIB
PE2#sh ip bgp vpnv4 vrf v1 labels | i 200.1.61.4
200.1.61.4/30 10.13.1.61
nolabel/25
PE2#
RR1
PE2#sh ip cef vrf v1 200.1.61.4 PE1
200.1.61.4/30, version 64, epoch 0, cached adjacency toAS#1
Serial2/0
0 packets, 0 bytes
Ser2/0
tag information
set
MPLS Backbone
200.1.61.4/30
local tag: VPN-route-headLoop0:10.13.1.61/32
fast tag rewrite with Se2/0, point2point, tags imposed: {2003 20}
CE1
via 10.13.1.61,
0 dependencies, recursive
next hop 10.13.2.5, Serial2/0 via 10.13.1.61/32
valid cached adjacency
tag rewrite with Se2/0, point2point, tags imposed: {2003 20}
PE2#
PE2
Loop0:10.13.1.62/32
CE-2
Fix: clear ip route vrf <vrf> <prefix>.

If the mismatch doesnt go away, then debug ip bgp vpn and debug
mpls lfib cef to dig in.
MPLS VPN Trouble
20

#8: Remote PE receives the route, but
remote CE doesnt
PE1
AS#65000
Ser2/0
CE1
AS#1
MPLS Backbone
Loop0:10.13.1.61/32
PE2
router bgp 1
!
address-family ipv4 vrf v1
neighbor 200.1.62.6 as-override
exit-address-family
!
CE-2
AS#65000
Loop0:10.13.1.62/32
Loop0:5.5.5.5/32
TIP: If eBGP on PE-CE and VPN sites use the same ASN, then configure
as-override on the BGP VRF af on both PEs
If IGP on PE-CE, then validate BGP->IGP redistribution (within IGP VRF) on
the PE
MPLS VPN Trouble
21
MPLS VPN Control Plane

Show Commands on PE
1.
sh ip bgp vpn all summary

Analogous to sh ip bgp summary; Lists all the MPBGP and CE
peers
2.
sh ip bgp vpn all

Lists all the VPN prefixes advertised/rcvd by the router
3.
sh ip bgp vpn vrf <vrf> summary

Similar to the first one, but for a specific VRF
4.

Lists all the VPN prefixes received in a specific VRF
5.
sh ip bgp vpn vrf <vrf> labels

List labels for the VPN prefixes in a VRF
MPLS VPN Trouble
22

Show Commands on PE
If OSPF on PE-CE sh ip ospf neighbors
Lists both VPN(s) and non-VPN(s) OSPF neighbors
sh ip ospf <process-id>
Select the VRF associated process-id to see relevant OSPF info (a
lot of info)
sh ip ospf <process-id> database

Select the VRF associated process-id to see the OSPF database for
that VRF
clear ip ospf <process-id>

Clear OSPF neighbors in the VRF if VRF associated process-id is
chosen
MPLS VPN Trouble
23

Show Commands on PE
If EIGRP on PE-CE
sh ip eigrp vrf <vrf> topology
Lists VRF specific EIGRP topology
sh ip eigrp vrf <vrf> neighbor|interface

Lists EIGRP neighbors or interfaces in the VRF
sh ip eigrp vrf <vrf> events

Shows VRF specific EIGRP events
clear ip eigrp vrf <vrf> neighbors

Clears VRF specific EIGRP neighbors
MPLS VPN Trouble
24

Clear Commands on PE
Relevant towards RR (or remote PE) peers:
clear ip bgp * vpnv4 unicast in
Route-refresh request is sent to all the
MP-BGP peers
clear ip bgp <MP-BGP peer> vpnv4

unicast in
Route-refresh request is sent to a specific
MP-BGP peer
MPLS VPN Trouble
25

Clear Commands on PE
Relevant towards CEs:
clear ip bgp * vrf < vrf >
Clear all PE-CE eBGP sessions in that vrf
clear ip bgp * vrf <vrf> in

Route-refresh message is sent to all the CEs in that vrf
clear ip bgp * vrf < vrf > out

Send respective VPN routes to all the CEs in that vrf
clear ip bgp <CE> vrf < vrf > soft in|out

soft reset of BGP session
MPLS VPN Trouble
26

Show Commands on RR
Route-reflector know nothing about VRF

Following commands come quite handy (especially on RR)
1.
sh ip bgp vpn all
2.
sh ip bgp vpn rd <RD>

Lists all VPNv4 prefixes that have RD in them
3.
sh ip bgp vpn rd <RD> label

Lists labels for VPNv4 prefixes that have RD
MPLS VPN Trouble
27

Debugs on PE
Be Careful on the Production Routers
1.
debug ip bgp vpnv4
Useful while troubleshooting label related

problems in BGP (could spit a lot of output)
2.
debug mpls lfib cef [acl]

Useful troubleshooting label mismatch in FIB/LFIB
3.
debug ip bgp vpnv4 import

Useful when VPN prefixes dont get imported in the VRF table
(could spit a lot of output)
4.
debug ip routing vrf <vrf> [acl]

Useful when VPN prefixes dont get installed in
the VRF routing table
MPLS VPN Trouble
28
Agenda
Control Plane
Real-life Examples
Forwarding Plane
Dissecting LFIB
Loadsharing in MPLS VPN Networks
Real-life Examples
Conclusion
MPLS VPN Trouble
29
MPLS VPN Forwarding Plane

Dissecting LFIB: show mpls forward
IP (or IGP) Prefix in the LFIB
RSP-PE-WEST-4#sh mpls forward 10.13.1.11 detail
Local Outgoing
Prefix
Bytes tag Outgoing
tag
tag or VC
or Tunnel Id
switched
interface
45
51
10.13.1.11/32
0
Fa1/1/1
MAC/Encaps=14/18, MRU=1500, Tag Stack{51}
0003FD1C828100044E7548298847 00033000
No output feature configured
Per-packet load-sharing
RSP-PE-WEST-4#
MRUMax Receivable Unit; The

Received Packet Will Be Transmitted
Unfragmented on Fa1/1/1, If Received
Packets Size Is Not More Than 1500B
Next Hop
10.13.7.33
Only One
Outgoing Label in
the Label Stack
MAC header = 0003FD1C828100044E754829

MPLS Ethertype = 0x8847
Label
= 0x00033000 = 51
0x00033000 = EXP+S
0x00033000 = MPLS TTL
Although MAC Header Is of 14 Bytes, Actual Encapsulation

I.E. MAC+MPLS Header Is of 18 Bytes (One Label Is 4 Bytes)
MPLS VPN Trouble
30

Dissecting LFIB: show mpls forward (Cont.)
PE1
VPN Prefix in the LFIB
P1
PE2
CE1
5.5.5.5/32
PE1#sh
Local
tag
27
mpls for vrf v1 5.5.5.5 detail

Outgoing
Prefix
Bytes tag
tag or VC
or Tunnel Id
switched
Untagged
5.5.5.5/32[V]
0
MAC/Encaps=0/0, MRU=1504, Tag Stack{}
VPN route: v1
No output feature configured
Per-packet load-sharing
PE1#
Outgoing
interface
Se2/0
Next Hop
point2point
Se2/0 Is a PE-CE
Interface which
Is under VRF v1
Only 1504 Byte Size Packet Can Be Received because

15044 (for One Label 27) = 1500 Is the MTU Size of Se2/0
MAC/Encaps Field Corresponds to the tag adj, and
because the VRF Interface Doesnt Typically Have MPLS
Enabled, tag adj Is 0; hence, 0/0 Output
MPLS VPN Trouble
31
Agenda
Control Plane
Real-life Examples
Forwarding Plane
Dissecting LFIB
Real-life Examples
Conclusion
MPLS VPN Trouble
32
MPLS VPN Fwd PlaneLoadsharing

Loadsharing in MPLS VPN network is same as that of
the IP network
i.e. FIB per-source-destination loadsharing
IP src and dest addresses inside the MPLS packet are

hashed to find the right LSP
Lets Go through PE-P and P-P Loadsharing

MPLS VPN Trouble
33
MPLS VPN Fwd PlaneLoadsharing (I)

PE-P Loadsharing (Cont.)
PE1#sh ip cef vrf v1 200.1.62.4
200.1.62.4/30, version 13, epoch 0, per-destination sharing
0 packets, 0 bytes
tag information set
local tag: VPN-route-head
fast tag rewrite with
Recursive rewrite via 10.13.1.62/32, tags imposed {25}
via 10.13.1.62, 0 dependencies, recursive
next hop 10.13.1.9, Ethernet1/0 via 10.13.1.62/32
valid adjacency
tag rewrite with
Recursive rewrite via 10.13.1.62/32, tags imposed {25}
Recursive load sharing using 10.13.1.62/32.
PE1#
Because There Are Loadshared Paths

to the Egress PE i.e. 10.13.1.62/32
PE1
E0/0
E1/0
P1
Se2/0
Loop0:10.13.1.62/32
PE2
Only VPN Label Is Shown
Dont panicIGP label is chosen during the

forwarding (depending on the hash-bucket)
MPLS VPN Trouble
34

PE-P Loadsharing (Cont.)
PE1#sh ip cef 10.13.1.62
0 packets, 0 bytes
tag information set, shared
local tag: 18
via 10.13.1.5, Ethernet0/0, 1 dependency
traffic share 1
next hop 10.13.1.5, Ethernet0/0
valid adjacency
tag rewrite with Et0/0, 10.13.1.5, tags imposed: {2001}
traffic share 1
valid adjacency
0 packets, 0 bytes switched through the prefix
tmstats: external 0 packets, 0 bytes
internal 0 packets, 0 bytes
PE1#
CE1
30.1.61.4/30
PE1
E0/0
E1/0
P1
Se2/0
Loop0:10.13.1.62/32
CE2
PE2
200.1.61.4/30
IGP Label Is Right Here
IGP Label and the outgoing interface are derived

after the hash-bucket is decided
MPLS VPN Trouble
35

PE-P Loadsharing (cont.)
CE1
30.1.61.4/30
PE1
PE1#sh ip cef vrf v1 exact-route 30.1.61.4 200.1.62.4 internal

30.1.61.4
-> 200.1.62.4
: Ethernet1/0 (next hop 10.13.1.9)
Bucket 7 from 16, total 2 paths
E0/0
E1/0
P1
Se2/0
Loop0:10.13.1.62/32
CE2
PE2
200.1.61.4/30
In summary, the show-output in load-sharing case

gets bit tricky; but the fundamental is the same
MPLS VPN Trouble
36
MPLS Fwd PlaneLoadsharing (II)

PE1
P-P Loadsharing
P1#sh mpls for 10.13.1.62
Local Outgoing
Prefix
tag
tag or VC
or Tunnel Id
52
21
10.13.1.62/32
27
10.13.1.62/32
P1#
Bytes tag
switched
0
0
Outgoing
interface
Eth0/0
Eth1/0
Next Hop
point2point
point2point
For VPN traffic, P router hashes the IP

src+dest to apply the packet to the correct
hash bucket
P1
E0/0
E1/0
P2
P3
Se2/0
Loop0:10.13.1.62/32
PE2
sh ip cef exact-route command cant be used on the P

router since it doesnt know the VPN addresses
Hence, rely on (LFIB) counters to make sure the traffic is getting
loadshared
MPLS VPN Trouble
37
Agenda MPLSVPN Troubleshooting

Control Plane
Real-life Examples
Forwarding Plane
Dissecting LFIB
Real-life Examples
Conclusion
MPLS VPN Trouble
38

Troubleshooting Tips
Symptom
Tip
Cisco IOS
Command
CE CE Traffic Fails
Verify That the

PE PE VPN Traffic Can Pass Using
vrf Pings (Assuming the Control Plane
Information Has Already Been Verified*)
PE PE MPLS
Traffic Fails
Validate the PE->PE IP Connectivity;

and then Check
for the LSP
PE#ping <remotePE>
PE PE IP Traffic Passes, but

MPLS
Traffic Fails
Find out where Exactly the LSP Is

Broken
PE#ping mpls ipv4

<remotePE> **
Incoming MPLS Traffic

Is Dropped at the
Egress PE
Check the LFIB Entries on Both RP, LC

(and Relevant HW Engines, if Present)
PE#sh mpls for | i

<prefix>
MPLS VPN Trouble
PE#ping vrf <vrf>

<remote prefix>
39

Troubleshooting Tips (Cont.)
Symptom
Tip
CE CE Traffic Fails for

Certain MTU Sizes
Check the MPLS MTU Size of the MPLS

Enabled Interfaces and Make Sure It Is
More than the Reported Failed MTU Size
CE CE Traffic Fails for

Certain MTU Sizes
Verify that the Ethernet Switch Ports

inside the MPLS Core Is Enabled with
Baby Giant Support
Cisco IOS
Command
Router#sh mpls int
de | in MTU
Switch#sh port jumbo
*Please See the Troubleshooting Control Plane TechTalk;

**12.0(26)S Onwards
MPLS VPN Trouble
40

Troubleshooting Steps
The VPN Traffic Outage Has Been Reported to You:
Step 1:
Step 2:
Step 3:
Step 4:
Step 5:
Step 6:
Step 7:
Step 8:
MPLS VPN Trouble
First, verify VRF ping from PE1 to PE2 (interface addresses)

If passed, then either CE PE or PE CE may be the problem =>
not a MPLS core problem; STOP and Check whether the packets
are getting dropped by ingress LC on PE
If failed, then MPLS core may be the problem; PROCEED
Ping ingress PE to egress PE to verify the IP reachability
If failed, then STOP and verify egress PEs route hop-by-hop
If passed, then traceroute PE1 PE2 and PE2 PE1 to ensure
the PE-to-PE LSP setup
Also check for the labels in each line of the traceroute output; if
traceroute fails for some reason, then STOP and verify the label at
each hop
If good, then the problem may be specific to the HW on either PE
or P routers; find out that whether the HW is dropping the packets
by looking at the interface counters by sh int <int>
41
MPLS VPN Fwd PlaneTroubleshotting

Lets do some more trouble(shooting)
MPLS VPN Trouble
42
MPLS VPN Fwd Plane - Troubleshooting

Problem: A VPN Site Cant Reach Other Sites
Cause: The CE CE Traffic Is Getting Dropped Somewhere
PE1
200.1.61.4/30
P1
E0/0
E1/0
Ser2/0
MPLS Backbone
Loop0:10.13.1.61/32
200.1.62.4/30
Loop0:10.13.1.62/32
CE1
CE2
Tip 1: Check the control plane information first
FIB
PE2
LFIB
PE1#sh ip cef vrf v1 200.1.62.4;
PE1#sh mpls for vrf v1 | inc 200.1.61.4
PE2#sh ip cef vrf v1 200.1.61.4;
Make sure that the label information is correct
Turn on deb ip icmp on both PEs

Step 1: Issue ping vrf v1 <remote_PE-CE_address> on both PEs
Step 2: If they pass, then we have verified that the problem is not in the
MPLS core

200.1.61.4/30
Ser2/0
CE1
5.5.5.5/32
PE1
P1
E0/0
E1/0
Ser2/0
PE2
200.1.62.4/30
MPLS Backbone
Loop0:10.13.1.61/32

200.1.62.4/30, version 10, epoch 0, per-destination
sharing
0 packets, 0 bytes
tag information set
fast tag rewrite with
Recursive rewrite via 10.13.1.62/32, tags imposed
{25}
next hop 10.13.1.9, Ethernet1/0 via 10.13.1.62/32
valid adjacency
tag rewrite with
Recursive rewrite via 10.13.1.62/32, tags imposed
{25}
Recursive load sharing using 10.13.1.62/32.
PE1#
Validated the Labels

in PE1->PE2 Direction
Loop0:10.13.1.62/32
CE-2
6.6.6.6/32

25 Aggregate 200.1.62.4/30[V] 0
PE2#
PE1#sh ip cef 10.13.1.62
0 packets, 0 bytes
tag information set
local tag: 18
traffic share 1
valid adjacency
via 10.13.1.9, Ethernet1/0, 2 dependencies
traffic share 1
valid adjacency
0 packets, 0 bytes switched through the prefix
PE1#

200.1.61.4/30
Ser2/0
CE1
5.5.5.5/32
PE1
P1
E0/0
E1/0
Ser2/0
PE2
200.1.62.4/30
MPLS Backbone
Loop0:10.13.1.61/32
PE1#sh mpls for vrf v1 | i 200.1.61.4

28 Aggregate 200.1.61.4/30[V] 0
PE1#
Loop0:10.13.1.62/32
6.6.6.6/32

200.1.61.4/30, version 73, epoch 0, cached adjacency to
Serial2/0
0 packets, 0 bytes
tag information set
fast tag rewrite with Se2/0, point2point, tags imposed:
{2003 28}
next hop 10.13.2.5, Serial2/0 via 10.13.1.61/32
valid cached adjacency
tag rewrite with Se2/0, point2point, tags imposed:
{2003 28}
PE2#
Validated the Labels in PE2 PE1 Direction

CE-2

200.1.61.4/30
Ser2/0
CE1
5.5.5.5/32
PE1
P1
E0/0
E1/0
Ser2/0
PE2
200.1.62.4/30
MPLS Backbone
Loop0:10.13.1.61/32
Loop0:10.13.1.62/32
PE1#deb ip icmp
ICMP packet debugging is on
PE1#
Step 1 PE1#ping vrf v1 200.1.62.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.1.61.5, timeout is
2 seconds:
.....
Success rate is 0 percent (0/5)
PE1#
CE-2
6.6.6.6/32
PE2#deb ip icmp
PE2#
PE2#
*May 11 00:42:16.353: ICMP: echo reply sent, src
200.1.62.5, dst 200.1.61.5
200.1.62.5, dst 200.1.61.5
200.1.62.5, dst 200.1.61.5
200.1.62.5, dst 200.1.61.5
200.1.62.5, dst 200.1.61.5
PE2#
Step 3: Okalthough the vrf pings failed at PE1, ICMP debugs at PE2
confirms that PE1->PE2 LSP is error free
Lets ping in the opposite direction to check the PE2 PE1 LSP

200.1.61.4/30
Ser2/0
CE1
5.5.5.5/32
PE1
P1
E0/0
E1/0
Ser2/0
Loop0:10.13.1.61/32
Loop0:10.13.1.62/32
Since PE1 didnt get/show any

ICMP echos for the vrf pings
Either PE2 PE1 LSP is broken
Or PE1 is dropping the received MPLS packets for

some reason
Okso lets troubleshoot for (a) first

CE-2
6.6.6.6/32
PE2#deb ip icmp
PE2#
PE2#ping vrf v1 200.1.61.5
Sending 5, 100-byte ICMP Echos to 200.1.61.5, timeout is
2 seconds:
.....
PE2#
b) Or PE1 doesnt have the LFIB entry for 200.1.61.5

c)
200.1.62.4/30
MPLS Backbone
PE1#deb ip icmp
PE1#
PE1#
PE1#
a)
PE2
We Already
Verified This
Earlier

200.1.61.4/30
Ser2/0
CE1
5.5.5.5/32
PE1
P1
E0/0
E1/0
Ser2/0
PE2
200.1.62.4/30
MPLS Backbone
Loop0:10.13.1.61/32
Loop0:10.13.1.62/32
Step 4 PE1#ping 10.13.1.62

Sending 5, 100-byte ICMP Echos to 10.13.1.62,
timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip
min/avg/max = 40/57/92 ms
PE1#
P1#sh mpls forward
Local Outgoing
tag
tag or VC
2003
Untagged
Untagged
P1#
CE-2
6.6.6.6/32
PE2#ping 10.13.1.61
Sending 5, 100-byte ICMP Echos to 10.13.1.61,
timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip
min/avg/max = 28/52/72 ms
PE2#
10.13.1.61
Prefix
or Tunnel Id
10.13.1.61/32
10.13.1.61/32
Bytes tag
switched
0
0
Outgoing
interface
Et0/0
Et1/0
Step 7: IP reachability is confirmed between PE1 and PE2 (steps 1 and 2);
GOOD; but that doesnt validate the LSP in both directions
Step 7: Per P1s LFIB, it doesnt have the right label to reach PE1 (untagged vs. Pop).
Next Hop
10.13.1.6
10.13.1.10

Rememberuntagged outgoing label means that
get rid of the label stack; hence, the VPN label would
be lost at P1
untagged label for /32 routes inside the MPLS core
is almost always bad
To fix this untagged problem in LFIB,
Check whether LIB and LFIB are in-sync about this entry.
If not, then clear ip route 10.13.1.61 on P1
If yes, then flap the LDP neighbor by clear mpls ldp
neighbor 10.13.1.61 on P1 to relearn the correct binding
** See more Debugs at the Show commands section

Till now, there wasnt any clean way of DETECTING
THE BROKEN LSPS
LSP pings* can DETECT the broken LSPs
LSP traceroute* can PIN-POINT the culprit router
where the LSP could be broken
But we will still have to fix the LSP
LSP ping would simply replace steps 6 and 7
in the troubleshooting steps
*12.0(26)S Onwards
MPLS VPN Forwarding Plane -Trouble #1 (Cont.)

PE1#ping mpls ipv4 10.13.1.62/32
Sending 5, 100-byte MPLS Echos to 10.13.1.62/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not transmitted,
'.' - timeout, 'U' - unreachable,
'R' - downstream router but not target
LSP Ping Failed

RRRRR
PE1#
PE1#ping mpls ipv4 10.13.1.62/32

Sending 5, 100-byte MPLS Echos to 10.13.1.62/32,
timeout is 2 seconds, send interval is 0 msec:
LSP Ping
Succeeded

!!!!!
PE1#
MPLS VPN Forwarding Plane -Trouble #1 (Cont.)

LSP traceroute* is capable of differentiating between
untagged and pop/null
PE1#trace mpls ipv4 10.13.1.62/32
Tracing MPLS Label Switched Path to 10.13.1.62/32, timeout is 2 seconds
0 10.13.1.10 MRU 1500 [Labels: 2002 Exp: 0]
R 1 10.13.2.14 MRU 1204 [No Label] 52 ms
! 2 10.13.2.14 52 ms
PE1#
PE1#trace mpls ipv4 10.13.1.62/32
Tracing MPLS Label Switched Path to 10.13.1.62/32, timeout is 2 seconds
0 10.13.1.10 MRU 1500 [Labels: 2002 Exp: 0]
R 1 10.13.2.13 MRU 1512 [implicit-null] 40 ms
! 2 10.13.2.14 68 ms
PE1#
*12.0(26)S Onwards
Agenda MPLSVPN Troubleshooting

Control Plane
Real-life Examples
Forwarding Plane
Dissecting LFIB
Real-life Examples
Conclusion
MPLS VPN Trouble
53
MPLS VPN Fwd PlaneShow Commands

sh mpls forwarding
Shows all LFIB entries (vpn, non-vpn, TE etc.)
sh mpls forwarding | inc <prefix>

Whether the prefix is present in the LFIB or not
sh mpls forwarding vrf <vrf> <prefix>

LFIB lookup based on a VPN prefix
sh mpls forwarding label <label>

LFIB lookup based on an incoming label
MPLS VPN Trouble
54
MPLS VPN Fwd PlaneShow Commands
sh ip arp vrf <vrf>

Lists ARP entries relevant to the <vrf> only
sh ip cef vrf <vrf > <prefix>

Displays the label stack, outgoing interface etc
sh mpls forwarding vrf <vrf>

Lists labels for the VPN prefixes learned from the CE(s)
MPLS VPN Trouble
55
MPLS VPN Fwd PlaneDebugs

Be Careful on the Production Routers
debug arp
Useful for VPN prefixes as well
debug mpls lfib cef [acl]

Useful when VPN prefixes have label mismatch among
BGP, FIB and LFIB.
MPLS VPN Trouble
56
Conclusion
MPLS seems cryptic, but it is not
Whether to look at FIB or LFIB?
Whether it is a BGP or MPLS problem?
Whether the problem is within the core or outside
the core?
Ongoing MPLS OAM work .
MPLS VPN Trouble
57
RST-3061
8186_05_2003_c1
2003, Cisco Systems, Inc. All rights reserved.
58

MPLS VPN Troubleshooting

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

MPLS VPN Troubleshooting

Uploaded by

Copyright:

Available Formats

Troubleshooting MPLS VPN Networks

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

MPLS VPN Trouble

2004 Cisco Systems, Inc. All rights reserved.

2004 Cisco Systems, Inc. All rights reserved.