BRKSPG-3612 Troubleshooting IOS-XR

Troubleshooting IOS-XR
BRKSPG-3612
Agenda
 Silver bullets
<QC> Quick Command
 Troubleshooting Tools
 Hardware
 Software
 New features
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
IOS-XR: Troubleshooting Tools
IOS-XR: Troubleshooting Tools
1. Logging
2. Trace (L-Trace and P-Trace)
3. Debug
4. Core dump / crash files
What is/has happening/ed?
 <QC>CRS#show ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
100.5.5.5 1 FULL/ - 00:00:35 100.100.10.20 POS0/5/1/0
 <QC>CRS#show logging | i 5-
%ROUTING-BGP-5-ADJCHANGE
%ROUTING-OSPF-5-ADJCHG
Trace command
 Always-on debug
 Reduces the need to enable debugging and wait for the
event to re-occur
 Need to be part of the “cisco-support” group
 Logs are cyclical and overwrite older events.
L-Trace command
 Lightweight Trace
 OSPF Troubleshooting
 Quick command
<QC>CRS#sh ospf trace all 10
 Will list the last 10 outputs of all the OSPF debugs
P-Trace command
 Packet Trace only on CRS

interface POS0/5/1/0
ipv4 address 100.100.10.10 255.255.255.0
capture software packets
 <QC>CRS#show captured packets egress interface

POS0/5/1/0 location 0/5/CPU0
 Will list the CPU punted traffic
Trace command
 BGP Troubleshooting
 <QC>CRS#sh bgp trace reverse
–Will list the BGP debugs in reverse order
 <QC>CRS#sh bgp trace reverse errors

–Will list the BGP errors in reverse order
Debug commands
 Always use “trace” before “debug”

 Quick commands
<QC>#debug bgp update
 Is a good way to see in real time the propagation of routing
updates/changes
Core Dumps
 Abnormally terminated processes will generate a core

dump
 <QC>CRS#show context
–This will show the last 10 core dumps (both crash and manually
triggered)
 <QC>CRS#show exception
–To see if writing a core dump failed
New Feature: Service Console – ASR9k
 <QC>ASR9K(admin)#service console
IOS-XR: Hardware
IOS-XR Hardware 12000 / GSR
Multi-Service Edge
(also runs IOS 12.0S)
ASR 9000
Aggregation Services
CRS
Core & MSE
CRS Logical Architecture – Packet Flow
IOS-XR Hardware CRS
 <QC>CRS#show sysmgr trace

–This will show where (HW) processes have been
started/stopped/restarted
 <QC>CRS(admin)#show platform
–This will show you what you have and is it running
 <QC>CRS(admin)#show inventory all

–This will show you all the hardware including serial numbers
Cisco CRS-1 Line Card and commands
MODULAR SERVICES CARD PLIM
To Fabric Ingress Packet Flow

OC192
Show
Ingress
interface [type] R/S/M
RX PSE
Interface
Module Framer
ASIC and
Queuing L3 Engine Optics
Show controllers plim asic statistics interface [type]OC192
1
R/S/M Framer
and
MIDPLANE
Squid CPU Optics
(FABRIC)
GW
Show controllers plim asic statistics summary OC192
Framer
location R/S/M and
Optics
From TX PSE Egress
Fabric L3 Engine Queuing OC192
ASIC Framer
and
Optics
From Fabric Egress Packet Flow
PLIM Packet Drops
 Clear plim stats before sending traffic.

 Verify that ingress plim "To PSE : Total Good Packets" are almost
equal to egress plim "From EgressQ: Total Good Packets".
 If BP count is incrementing "From PSE: Total BP Count", make
sure that LC is not oversubscribed.
 BP packets are sent to slowdown the traffic.
PLIM Packet Drops
 Clear ingress pse counters using below command before sending

traffic.
 The "Punt stats" are packets that are punted to CPU. All punted
packets are rate limited, above this rate limit dropped packets are
shown as "Policed & Dropped" "Drop stats" gives the brief reason
of why packets are dropped. Corresponding dropped counts are
shown.
PLIM Packet Drops
PLIM Packet Drops

Interface OC192
Module Framer
Ingress RX PSE and
ASIC
2
OC192
1 Framer
and
MIDPLANE
Show
Squidcontroller pse tcam summary
CPU Optics
(FABRIC)
GW
OC192
Framer
Show controller pse util and
Optics
FromTX PSE Egress
Show controller
Fabric
ASIC
pse stats
L3 Engine Queuing OC192
Framer
and
Optics

Interface OC192
Module Framer
Ingress RX PSE and
ASIC
3 2
OC192
1 Framer
and
MIDPLANE
Squid Optics
Show
GW controllers ingressq statistics
CPU
(FABRIC)
OC192
Framer
Show controllers ingressq queue all and
Optics
From TX PSE Egress
Fabric
Show controllers ingressq block dps
ASIC
L3 Engine Queuing OC192
Framer
and
Optics
Ingress Queue Packet Drops
Ingress Queue Packet Drops
 In a normal scenario, none of the "Ingressq Drops" should be
incrementing.
 In the above ouput "discard drops" will be incremented when
CRAB is oversubscribed , this is for only Unicast traffic.
 "tail drops " These are the threshold drops in the destination
queues of the SEAL for Unicast traffic.
 For multicast traffic check for "mcl tail drops " for low priority and
"mch tail drops" for high priority traffic.

4 Interface OC192
Module Framer
Ingress RX PSE and
ASIC
3 2
OC192
1 Framer
and
MIDPLANE
Squid
Show
GW controllers fabricq statistics
CPU Optics
(FABRIC)
OC192
Framer
Show controllers fabricq queues and
Optics
From TX PSE Egress
ASIC Framer
and
Optics
Fabric Queue Packet Drops

4 Interface OC192
Module Framer
Ingress RX PSE and
ASIC
3 2
Show controllers pse statistics egress location OC192
1 Framer
R/S/M and
MIDPLANE
Squid CPU Optics
(FABRIC)
GW
OC192
Show controllers egressq statistics location R/S/M
Framer
and
Optics
From
8
TX PSE Egress
ASIC 6 7 Framer
and
5 Optics
Egress PSE Drops
Egress Queue Drops
Egress PLIM Drops
 Verify that ingress plim "To PSE : Total Good Packets" are almost
equal to egress plim "From EgressQ: Total Good Packets".
 If BP count is incrementing "From PSE: Total BP Count", make
sure that ipogo is not oversubscribed.
 BP packets are sent from ipogo to inform the Beluga to slowdown
the traffic.
Egress PLIM Drops
IOS-XR: Software
Router OS Evolution
Control Plane Applications Control Plane Data Plane Management Plane

Control Plane Data Plane Management Plane
SSH
Forwarding Plane Applications
Control Plane Data Plane Management Plane
SSH
Drivers
Management Plane
Routin
NetFlo
Interfa
Per.fM
SNMP
Alarm
Service
OSPF
LPTS
IGMP
BGB
SSH
HA Infrastructure
QoS
ACL
XML
ISIS
gmt
PIM
RIB
FIB
RIP
PFI
CLI
Host
L2
ce
Network Stack
w
g
Applications
System Forward Checkpoint DB Multicast IPC System DB

Infrastructure Infrastructure Distributed Infrastructure
Scheduler Synch. Services IPC Mech Memory Mgmt

OS Scheduler
Kernel System Services
 Monolithic Kernel  Micro Kernel
 Centralised Infrastructure  Distributed Infrastructure
 Integrated Network stack  Independent Network stack
 Centralised applications  Distributed applications

IOS-XR Software
 Process restart
 Failed processes restart automatically – no system crash
 Enables modular software – add new features while running
 Enables modular repair – patches in between full releases
 Enables in-service upgrades (future)
 Protected process memory
 One process can’t write to another’s memory
 Pre-emptive multi-tasking
 DLLs – only required code is loaded
Terms
PIE Mini
SMU Package
Bundles
Security Manageability
MPLS Multicast
Optional MPLS, UCP PIM, MFIB, IGMP
IPSec, Encryption, ORB, XML,
Decryption Alarms management
Core Routing:
[Mandatory] RIB, BGP, ISIS, OSPF, RPL
Forwarding Line Card

Platform independent Platform Dependent
FIB, ARP, QoS, ACL, etc LC ucode & drivers
Base
Admin
Interface manager,
Resource Management:
System database, checkpoint services
Rack, Fabric, LR management
Configuration management, etc.
OS:
Kernel, file system, memory management, and other slow changing core
IOS XR Modular Packaged Software
RP DRP LC
Manage- Manage- Multi-cast Opt’l
Security Security GMPLS
ability ability
Opt’l Opt’l
GMPLS Multi-cast GMPLS Multi-cast
Line Card
Forwarding
RPL BGP RPL BGP Mand
OSPF OSPF Base
ISIS ISIS
Forwarding Forwarding OS Mand
Mand Mand
Base Base
SC
Admin OS
Base
OS
Admin Mand
OS
 Upgrade specific packages/Composites

 Point Fix for software faults
PIE
 PIE - Package Installation Envelopes. A delivery
mechanism for packages.
 Non-bootable files.
 Contain a single package or a set of packages, called a
composite package or bundle.
 Being non-bootable, they are used to add software
package files to a running router.
 PIE files have a pie extension.
SMU - Software Maintenance Upgrades
 When a PIE file contains software for a specific bug fix, it is
called a SMU.
 A composite SMU is a PIE file that contains SMUs for more
than one package.
 SMU versions are synchronised to the package major,
minor, and maintenance versions they upgrade. Eg. of a
SMU Name: hfr-rout-3.6.1.CSCsr70080
 Not an alternative to maintenance releases. All bugs fixed
by SMUs are integrated into the maintenance releases.
IOS-XR Software Naming
 comp - The “comp” prefix indicates that the file is a
composite of multiple packages.
 platform - Platform for which the software package is
designed. The platform designation is “hfr.”
 composite_name - Identifies a specific composite package.
Only composite PIE file is named “mini” and includes
packages described Unicast Routing Core Bundle.
 package_type – Type of package the file supports. Applies
only to single-package PIEs. Example – fwdg
[Forwarding package] , k9sec [Security package].
IOS-XR Software Naming (cont.)
 major - Identifies the major release of this package.
–All packages on the router must be at the same major release level.
May require a router reboot
 minor - Contains one new features and/ or bug fixes.
–Need not be identical for all packages on the router but has to be
compatible. A minor release may require a router reboot.
 maintenance - Contains a collection of bug fixes for a
package.
–Need not be identical for all packages on the router but the major
and minor versions of the maintenance release must match. Does
not usually require a router reboot.
IOS-XR Software Naming (cont.)
Software Delivery
Filename Example
Type
comp-platform-
Composite (Bundle) PIE composite_name.pie- comp-hfr-mini.pie-3.7.0
major.minor.maintenance
platform-package_type.-p.pie- hfr-mgbl-p.pie-3.7.0
Single package PIE
major.minor.maintenance hfr-mpls-p.pie-3.2.0
comp-platform-
Composite SMU
composite_name.ddts.pie
comp-hfr-001.CSCec98xxx.pie
platform-package_type-
Single package SMU
major.minor.maintenance.ddts.pie
hfr-fwdg-3.7.0.CSCei4xxx.pie
IOS-XR Upgrade
 Installing XR software from ROM Monitor mode required
“vm files”.
 PIEs cannot be used to install XR software from ROM
Mon.
 VM files are used for router recovery when the router
cannot boot the Cisco IOS XR software.
 VM files contain the same packages as the Unicast Core
bundle pie. The format of a vm file is:
comp-platform-package_name.vm-major.minor.maintenance
Example - comp-hfr-mini.vm-3.9.0
IOS-XR Upgrade (cont.)
 The tar files available on CCO contain both PIE files and
VM files. The format of a tar file is:
platform-bundle_name-major.minor.maintenance.tar
Example – CRS-1-iosxr-3.8.0.tar
 bundle_name - 2 bundles available:
-- IOS XR - Unicast Routing Core Bundle, Management,
MPLS, and Multicast packages.
-- IOS XR-k9 – IOS XR bundle file plus the Security
package.
IOS-XR Upgrade Summary
 Initial Installation is always from ROMMON
 Sub-sequent upgrades can be done from CLI or ROMMON
 In ROMMON, Use “.vm” file
 From CLI, Use “.pie” file
 SMU’s and security packages can only be added from CLI
IOS-XR Process
 Copy the package file or files to a local storage device or
file server.
 Add the package or packages on one or more SDRs using
the command install add.
 Activate the package or packages on one or more SDRs
using the install activate command.
 Commit the current set of packages using the install
commit command.
IOS-XR Upgrade Process
IOS-XR Rollback
 Each time a package is activated or deactivated, a
rollback point is created.
 This point defines the package set that was active at that
point.
 A rollback point is also there for the last committed
package set.
 <QC>#show install rollback to [id]
 <QC>#show install request
IOS-XR Always Check
 Check the recommended upgrading procedure on CCO:
 http://www.cisco.com/web/Cisco_IOS_XR_Software/index.
html
 Make sure you have the correct rommon version on all line
cards and RP’s
 <QC>#(admin)#show hw-module fpd location all
Pre-emptive multi-tasking, multi-threaded
Multi-threading on CRS-1
 <QC>CRS#show process snmpd
Multi-threading on the CRS-1
 <QC>CRS#show processes threadname 378

–This will give us a list of the threads and their current status
Multi-tasking on CRS-1
 Use the UNIX utility “top” to obtain information regarding

current running tasks on the CRS-1
 <QC>CRS#top
Troubleshooting High CPU Utilisation
 <QC>#show processes cpu

 <QC>#run top_procs
 <QC>#show processes blocked
 <QC>#show processes <process name>”
 <QC>#follow process <process-id>”
 <QC>#dumpcore running <process job-id>
Virtual Memory, protected address space
Virtual Memory on CRS-1
 <QC>CRS#show process memory 63
Blocked Processes
 Blocked processes are a natural part of operation for a

multi-tasking, multi-threaded operating system
 Software caveats can, however, cause processes or
threads to remain in a blocked state resulting in
deadlocks.
 There are two common types of process blocks:
 Mutual Exclusion block (threads within process)
 Send/Reply/Receive block (between processes)
Mutual Exclusion block
 Mutual Exclusion (mutex) block

Thread is blocked on another thread of the same process
Mutual Exclusion block
 Mutual Exclusion (mutex) block
Send/Reply/Receive block
 Send/Reply/Receive block (Inter Process)

Process is blocked in Send/Reply or Receive state waiting on a
different process
Send/Reply/Receive block
 <QC>CRS#show processes blocked

–Identifying process in Send/Reply or Receive blocking state on the
CRS-1
Troubleshooting
 High CPU Utilisation /Memory Leak

 Important: Know your baseline
 Which processes are normally blocked
 What is the CPU/memory utilisation normally
Troubleshooting Blocked Processes

 <QC>CRS#show processes proc_name
 <QC>CRS#follow process proc_id
 <QC>CRS#dumpcore running job_id
“In routing, the control plane is the part of
architecture that is concerned with drawing the
network map, or more simply the information that
defines what to do with incoming packets.”
Definition: Control Plane
Control Plane Troubleshooting
 In-band
 Contention
 ASIC
 QoS
Control Plane interactions
Routing Routing
RIB Updates/
Process
Control Adjacencies
Plane
MPLS Label Binding
LIB Updates/
Process
Adjacencies
Forwarding MFI FIB

Plane
MPLS Traffic IP Traffic
Forwarding Forwarding
QNX
 QNX – Microkernel-based OS containing:

–CPU Scheduling
–InterProcess Communication (IPC)
–Interrupt redirection
 Almost everything else runs as user processes, including

all the Cisco IOS code
 KSH Shell access providing access to generic UNIX
tools/utilities
Troubleshooting BGP
RIB
BGP Table
BGP Best Path
The Usual Suspects
 Configs;
 Layer 1; IGP; Design;
 CPU; MTU; Memory
 Input Queues
 Bug
 ……….
BGP Flowchart
www.cisco.com/en/US/tech/tk365/technologies_tech_note09
186a008009478a.shtml#main
Configs
 <QC>CRS#show run router bgp neighbor-group internal

router bgp 100
bgp router-id 100.2.2.2
neighbor-group internal
remote-as 100
update-source Loopback100
address-family ipv4 unicast
neighbor 100.4.4.4
use neighbor-group internal
L1
 <QC>CRS#ping 100.4.4.4 source 100.2.2.2

 <QC>CRS#show tcp brief
Mon Feb 7 22:50:21.992 AEST
PCB VRF-ID Recv-Q Send-Q Local Address Foreign Address State
0x500fd8d0 0x60000000 0 0 100.2.2.2:179 100.4.4.4:42852 ESTAB
0x500fda78 0x60000000 0 0 100.4.4.4:59603 100.2.2.2:179 ESTAB
Trace command
 <QC>CRS#show bgp trace reverse

–Will list the BGP debugs in reverse order
 <QC>CRS#show bgp trace reverse errors

–Will list the BGP errors in reverse order
 <QC>CRS#show bgp trace ?

–There are a lot of options
Debug commands
 Always use “trace” before “debug”

 <QC>CRS#debug bgp update
 Is a good way to see in real time the propagation of routing
updates/changes
Process Debugging
 The “sysmgr” is foundation and parent process of all

processes and is responsible for the start/stop/restart of all
processes
 There are two instances running in hot-standby on each
Node
 <QC>CRS#show processes log location all | i bgp
Memory
 <QC>CRS#show watchdog memory-state

Memory information:
Physical Memory: 4096 MB
Free Memory: 2124.82 MB
Memory State: Normal
Check Memory State is: Normal
CPU
 <QC>CRS#top
303 processes; 1425 threads;
CPU states: 97.7% idle, 1.4% user, 0.7% kernel
Memory: 4096M total, 2123M avail, page size 4K
JID TID PRI STATE HH:MM:SS CPU COMMAND

1 2 10 Run 0:02:30 0.76% procnto-600-smp-instr
464 3 10 Rcv 0:06:25 0.45% wdsysmon
65839 1 10 Rply 0:00:00 0.09% top
339 3 10 Rcv 0:05:04 0.09% netio
137 12 10 Rcv 0:01:09 0.08% bgp
Blocked Process

Checking for any clues – State and TimeInState
 <QC>CRS#show processes blocked extended location all |
i bgp
Blocked Process
 <QC>CRS#show processes blocked extended location all |

i bgp
Check if BGP is being blocked – follow the chain up
Jid Pid Tid Name State TimeInState Blocked-on
65548 12300 1 ksh Reply 159:28:23:0203 12296 devc-conaux
Collecting information
 <QC>CRS#show tech-support routing bgp

 <QC>CRS#show logging
 <QC>CRS#show install summary
 <QC>CRS#show version brief
What Next
 Collect what you are seeing

 Collect what you have done
 Call the TAC
Q&A
Complete Your Online Session Evaluation
Complete your session evaluation:
 Directly from your mobile device by visiting
www.ciscoliveaustralia.com/mobile and login
by entering your badge ID (located on the
front of your badge)
 Visit one of the Cisco Live internet stations

located throughout the venue
 Open a browser on your own computer to

access the Cisco Live onsite portal

BRKSPG-3612 Troubleshooting IOS-XR

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKSPG-3612 Troubleshooting IOS-XR

Uploaded by

Copyright:

Available Formats

Troubleshooting IOS-XR

 <QC>CRS#show ospf neighbor

100.5.5.5 1 FULL/ - 00:00:35 100.100.10.20 POS0/5/1/0

 Packet Trace only on CRS

 <QC>CRS#show captured packets egress interface

 <QC>CRS#sh bgp trace reverse errors

 Always use “trace” before “debug”

 Abnormally terminated processes will generate a core

 <QC>CRS#show sysmgr trace

 <QC>CRS(admin)#show inventory all

To Fabric Ingress Packet Flow

 Clear plim stats before sending traffic.

 Clear ingress pse counters using below command before sending

To Fabric Ingress Packet Flow

To Fabric Ingress Packet Flow

To Fabric Ingress Packet Flow

To Fabric Ingress Packet Flow

Control Plane Applications Control Plane Data Plane Management Plane

System Forward Checkpoint DB Multicast IPC System DB

Scheduler Synch. Services IPC Mech Memory Mgmt

Kernel System Services

 Monolithic Kernel  Micro Kernel

 Centralised Infrastructure  Distributed Infrastructure

 Integrated Network stack  Independent Network stack

 Centralised applications  Distributed applications

Forwarding Line Card

 Upgrade specific packages/Composites

 <QC>CRS#show process snmpd

 <QC>CRS#show processes threadname 378

 Use the UNIX utility “top” to obtain information regarding

 <QC>#show processes cpu

 <QC>CRS#show process memory 63

 Blocked processes are a natural part of operation for a

 Mutual Exclusion (mutex) block

 Mutual Exclusion (mutex) block

 Send/Reply/Receive block (Inter Process)

 <QC>CRS#show processes blocked

 High CPU Utilisation /Memory Leak

 <QC>CRS#show processes blocked

Forwarding MFI FIB

 QNX – Microkernel-based OS containing:

 Almost everything else runs as user processes, including

 <QC>CRS#show run router bgp neighbor-group internal

 <QC>CRS#ping 100.4.4.4 source 100.2.2.2

PCB VRF-ID Recv-Q Send-Q Local Address Foreign Address State

0x500fd8d0 0x60000000 0 0 100.2.2.2:179 100.4.4.4:42852 ESTAB

0x500fda78 0x60000000 0 0 100.4.4.4:59603 100.2.2.2:179 ESTAB

 <QC>CRS#show bgp trace reverse

 <QC>CRS#show bgp trace reverse errors

 <QC>CRS#show bgp trace ?

 Always use “trace” before “debug”

 The “sysmgr” is foundation and parent process of all

 <QC>CRS#show watchdog memory-state

Check Memory State is: Normal

JID TID PRI STATE HH:MM:SS CPU COMMAND

 <QC>CRS#show processes blocked

 <QC>CRS#show processes blocked extended location all |

 <QC>CRS#show tech-support routing bgp

 Collect what you are seeing

 Visit one of the Cisco Live internet stations

 Open a browser on your own computer to

You might also like