Troubleshooting Forwarding Plane

Telefnica Espaa Support Advanced Services EMEA sas-te@juniper.net V1.1 20110527

INTERFACE AND BYPASS-ROUTING

2
Copyright 2009 Juniper Networks, Inc. www.juniper.net

PING TO REMOTE ADDRESS DEFAULT

Assuming default configuration (without default-address-selection)

RE CB FPC #1
PIC 1/0
(SA, DA) = (10.2.2.1, 10.2.2.2)

user@m320> ping 10.2.2.2

RE CB

PIC 4/1

FPC #4

FEB #4

(10.2.2.2, 10.2.2.1)

SIBs

so-1/0/0

10.2.2/30 10.3.3/30

so-4/1/0 .2 so-5/1/0
www.juniper.net

M320

.1 so-1/1/0

M120

Copyright 2009 Juniper Networks, Inc.

LINK KEEPALIVES & ROUTING PROTOCOL PACKETS

HDLC/OAM/LMI keepalives follow the same hardware path as ping

RE CB FPC #1
PIC 1/0

RE CB
PIC 4/1
FPC #4

FEB #4

SIBs

so-1/0/0

10.2.2/30 10.3.3/30

so-4/1/0 .2 so-5/1/0
www.juniper.net

M320

.1 so-1/1/0

M120

Copyright 2009 Juniper Networks, Inc.

PING TO REMOTE ADDRESS INTERFACE OPTION

The interface option only alters the source IP address by default Similar to source option (monitor traffic interface displays packets)
RE CB
FPC #1 user@m320> ping 10.2.2.2 interface so-1/1/0 FEB #4

RE CB
FEB #5 FPC #5 PIC 5/1

PIC 1/1

PIC 1/0

(SA, DA) = (10.3.3.1, 10.2.2.2) (10.2.2.2, 10.3.3.1)

FPC #4 PIC 4/1

so-1/0/0

10.2.2/30 10.3.3/30

so-4/1/0 .2 so-5/1/0
www.juniper.net

M320

.1 so-1/1/0

M120

Copyright 2009 Juniper Networks, Inc.

PING TO REMOTE ADDRESS BYPASS-ROUTING

bypass-routing allows to force the packet to go out a given interface Only works properly at SONET/SDH interfaces
RE CB FPC #1
user@m320> ping 10.2.2.2 interface so-1/1/0 bypass-routing PIC 1/1
(SA, DA) = (10.3.3.1, 10.2.2.2)

RE CB
FPC #5

PIC 5/1

FEB #5

(10.2.2.2, 10.3.3.1)

SIBs

so-1/0/0

10.2.2/30 10.3.3/30

so-4/1/0 .2 so-5/1/0
www.juniper.net

M320

.1 so-1/1/0

M120

Copyright 2009 Juniper Networks, Inc.

PING TO LOCAL ADDRESS DEFAULT

By default, ping to local address does not leave the RE

Checked with show chassis ethernet-switch statistics
(10.2.2.1, 10.2.2.1)

RE CB
(10.2.2.1, 10.2.2.1)

user@m320> ping 10.2.2.1

RE CB

FPC #1

PIC 1/0

PIC 4/1

FPC #4

FEB #4

M320

so-1/0/0 .1

10.2.2/30

so-4/1/0 .2

M120

Copyright 2009 Juniper Networks, Inc.

www.juniper.net

PING TO LOCAL ADDRESS INTERFACE

The interface option only alters the source IP address by default

Still packet does not leave the Routing Engine
(10.2.2.1, 10.2.2.1)

RE CB
(10.2.2.1, 10.2.2.1)

user@m320> ping 10.2.2.1 interface so-1/0/0

RE CB

FPC #1

PIC 1/0

PIC 4/1

FPC #4

FEB #4

M320

so-1/0/0 .1

10.2.2/30

so-4/1/0 .2

M120

Copyright 2009 Juniper Networks, Inc.

www.juniper.net

PING TO LOCAL ADDRESS BYPASS-ROUTING

bypass-routing allows to force the packet to go out a given interface Only works properly at SONET/SDH interfaces
(10.2.2.1, 10.2.2.1)

RE CB FPC #1

user@m320> ping 10.2.2.1 interface so-1/0/0 bypass-routing PIC 1/0

(10.2.2.1, 10.2.2.1)

RE CB
FPC #4

PIC 4/1

FEB #4

M320

so-1/0/0 .1

10.2.2/30

so-4/1/0 .2

M120

Copyright 2009 Juniper Networks, Inc.

www.juniper.net

LOCAL AND REMOTE LOOPBACK

10
Copyright 2009 Juniper Networks, Inc. www.juniper.net

PING TO REMOTE ADDRESS LOOPBACK REMOTE

Packet loops until TTL expires

The RE originating ICMP echo packets receive ICMP time exceeded On the right: packet copies sent to the PFE hit firewall filters (counting)
(10.2.2.1, 10.2.2.2)

RE CB FPC #1

user@m320> ping 10.2.2.2

RE CB

PIC 1/0
(10.2.2.1, 10.2.2.2)

PIC 4/1

FPC #4

FEB #4

M320
user@m320> ping 10.2.2.2 PING 10.2.2.2 (10.2.2.2): 56 data bytes 36 bytes from 10.2.2.1: Time to live exceeded Vr HL TOS Len ID Flg off TTL Pro cks Src 4 5 00 0054 8212 0 0000 01 01 1f91 10.2.2.1
11
Copyright 2009 Juniper Networks, Inc. www.juniper.net

M120

Dst 10.2.2.2

PING TO REMOTE ADDRESS LOOPBACK LOCAL (I)

Packet loops until TTL expires

The RE originating ICMP echo packets receive ICMP time exceeded
user@m320> ping 10.2.2.2 user@m120> ping 10.2.2.1

RE CB FPC #1
(10.2.2.1, 10.2.2.2) (10.2.2.2, 10.2.2.1)

RE CB FEB #4

PIC 1/0

PIC 4/1

FPC #4

M320

M120

user@M320# edit interfaces so-1/0/0 [ no-keepalives ; sonet-options loopback local; ] user@M120# edit interfaces so-4/1/0 [ no-keepalives ; sonet-options loopback local; ]

(*) May be necessary to remove family iso and family mpls for the test
12
Copyright 2009 Juniper Networks, Inc. www.juniper.net

PING TO REMOTE ADDRESS LOOPBACK LOCAL (II)

Output firewall filters require double lookup and fabric pass

The RE originating ICMP echo packets receive ICMP time exceeded
user@m320> ping 10.2.2.2 user@m120> ping 10.2.2.1

RE CB FPC #1
(10.2.2.1, 10.2.2.2) (10.2.2.2, 10.2.2.1)

RE CB FEB #4

PIC 1/0

PIC 4/1

FPC #4

SIBs
[edit firewall family inet filter prueba-loopback] term unico then { count paquetes; accept; } [edit interfaces so-1/0/0 unit 0 family inet] filter output prueba-loopback;
13
Copyright 2009 Juniper Networks, Inc. www.juniper.net

PING TO LOCAL ADDRESS LOOPBACK REMOTE

Two simultaneous troubleshooting paths

Original packet looped by the remote PIC and sent back to originator On the right: packet copies sent to the PFE hit firewall filters (counting)
(10.2.2.1, 10.2.2.1)

RE CB FPC #1

user@m320> ping 10.2.2.1 interface so-1/0/0 bypass-routing PIC 1/0

(10.2.2.1, 10.2.2.1)

RE CB
FPC #4

PIC 4/1

FEB #4

M320

M120

user@M320# set interfaces so-1/0/0 no-keepalives user@M120# set interfaces so-4/1/0 no-keepalives user@M120# set interfaces so-4/1/0 sonet-options loopback remote
14
Copyright 2009 Juniper Networks, Inc. www.juniper.net

PING TO LOCAL ADDRESS LOOPBACK LOCAL

bypass-routing allows to force the packet to go out a given interface Only works properly at SONET/SDH interfaces
user@m320> ping 10.2.2.1 interface so-1/0/0 bypass-routing user@m120> ping 10.2.2.2 interface so-4/1/0 bypass-routing

RE CB FPC #1
(10.2.2.1, 10.2.2.2) (10.2.2.2, 10.2.2.1)

RE CB FEB #4

PIC 1/0

PIC 4/1

FPC #4

M320

M120

user@M320# edit interfaces so-1/0/0 [ no-keepalives ; sonet-options loopback local; ] user@M120# edit interfaces so-4/1/0 [ no-keepalives ; sonet-options loopback local; ]
15
Copyright 2009 Juniper Networks, Inc. www.juniper.net

IMPLEMENTATION DETAILS
16
Copyright 2009 Juniper Networks, Inc. www.juniper.net

IMPLEMENTATION DETAILS BYPASS-ROUTING

With a logical loop, the packet traverse both PIC framers

This would spot interoperability issues between the framers Problem can be isolated to be caused by the line or by the endpoints Not necessarily by which of the endpoints
user@m320> ping 10.2.2.1 interface so-1/0/0 bypass-routing

RE CB FPC #1
PIC 1/0
(10.2.2.1, 10.2.2.1)

RE CB
PIC 4/1
FPC #4

FEB #4

M320
17

SONET FRAMERS
Copyright 2009 Juniper Networks, Inc. www.juniper.net

M120

IMPLEMENTATION DETAILS LOOPBACK

The PIC just loops the SONET frame

The PIC framers do not modify the SONET frame at all There is no way with loopbacks to traverse both PIC framers
user@m320> show interfaces so-1/0/0 extensive | match trace Received path trace: m320 so-1/0/0 Transmitted path trace: m320 so-1/0/0

RE CB FPC #1
PIC 1/0

loopback local at M120 so-4/1/0 loopback remote at M120 so-4/1/0

RE CB FEB #4

PIC 4/1
(10.2.2.1, 10.2.2.2)

FPC #4

M320
18

SONET FRAMERS
Copyright 2009 Juniper Networks, Inc. www.juniper.net

M120

IMPLEMENTATION DETAILS TRANSIT PING

The record-route option is useful to spot fabric failures

Different hardware path followed for each type of packet

RE CB (control) FEB #5
PIC 4/0
FPC #4 FPC #5

5/1 PIC

CB (fabric)
transit ping with record-route option transit ping with no special option
19
Copyright 2009 Juniper Networks, Inc. www.juniper.net

FEB #4

NON-SONET INTERFACE CAPABILITIES

The bypass-routing option can be used, but it does not work The remote loopback option is not available either How to use loops? ping the remote link address, and count TTL expired packets
user@m320> show system statistics icmp | match exceed time exceeded: 177

loopback mode Interface Type SONET/SDH GE/100GE ATM FR (E3 IQ)

20

ping options interface Yes Yes Yes Yes

www.juniper.net

local Yes Yes Yes Yes

remote Yes No No No

interface & bypass-routing Yes No No No

Copyright 2009 Juniper Networks, Inc.

TOS OPTION
21
Copyright 2009 Juniper Networks, Inc. www.juniper.net

TOS VALUES DIFFERENT ENCODINGS

The table below displays the formats used for:
3-bit & 6-bit bin: inet-precedence & dscp classifiers & rewrite-rules 3-bit & 6-bit dec: from precedence & dscp | traffic-class firewall filters 8-bit dec: ping command tos option, both for IPv4&IPv6 8-bit hex: dscp or traffic class field displayed in tcpdump decoding

IP Precedence 3 bit bin 000 001 010 011 100 101 110 111
22

DSCP 6 bit 8 bit dec 0 8 16 24 32 40 48 56 bin 00000000 00100000 01000000 01100000 10000000 10100000 11000000 11100000
www.juniper.net

dec 0 1 2 3 4 5 6 7

bin 000000 001000 010000 011000 100000 101000 110000 111000

dec 0 32 64 96 128 160 192 224

hex 0x00 0x20 0x40 0x60 0x80 0xa0 0xc0 0xe0

Copyright 2009 Juniper Networks, Inc.

ICMP AS CONTROL TRAFFIC

Control traffic COS is determined by the RE The Routing Engine sets the DSCP/IP Precedence as well as the internal FC+PLP values of a packet before sending it to the Egress PFE By default, locally originated ICMP goes to queue 0
Regardless of the ping tos value

The ping tos option can change the DSCP/IP Precedence but not the queue the packet goes to The ICMP echo reply mirrors the DSCP/IP Precedence from the original ICMP echo request In Junos OS 10.4 output lo0 firewall filters support actions to rewrite FC,PLP (queue number) and DSCP/IP Precedence independently before sending packet to PFE Egress control packets are never processed by rewrite rules
23
Copyright 2009 Juniper Networks, Inc. www.juniper.net

LAB DIAGRAMS
24
Copyright 2009 Juniper Networks, Inc. www.juniper.net

NETWORK DIAGRAM
lo0.0 10.100.3.3

M7i

lo0.0 10.100.1.1 so-1/0/0 10.2.2/30 10.3.3/30 so-4/1/0 .2 so-5/1/0

lo0.0 10.100.2.2

M320

.1 so-1/1/0

M120

25

Copyright 2009 Juniper Networks, Inc.

www.juniper.net

FAILURE SCENARIO COMPLETE TRAFFIC LOSS IN A LINK

test failure test success

RE CB FPC #1
PIC 1/0

RE CB
PIC 4/1
FPC #4

FEB #4

M320

M120

26

Copyright 2009 Juniper Networks, Inc.

www.juniper.net

FAILURE SCENARIO TRAFFIC DEGRADATION IN A SINGLE LINK

M320
RE CB FPC #1
PIC 1/1

M120
RE CB
PIC 5/1
FPC #5

FEB #4

FEB #5 FPC #4
PIC 1/1 PIC 4/1

PIC 4/0

test failure test success

27

Copyright 2009 Juniper Networks, Inc.

www.juniper.net

FAILURE SCENARIO TRAFFIC DEGRADATION IN A DOUBLE LINK

M320
RE CB
FPC #5

M120
RE CB
PIC 1/1

FPC #1

PIC 5/1

FEB #4

FEB #5 FPC #4
PIC 4/1

PIC 4/0

PIC 1/0

test failure test success

M7i

28

Copyright 2009 Juniper Networks, Inc.

www.juniper.net

CHANGE-LOG: When 20110526 20110527 Who Rev What

amonge@juniper.net v1.0 Presented to customer amonge@juniper.net v1.1 Added lab slides, sending to customer