Professional Documents
Culture Documents
watch for:
Where the circuit resides?
What interface its connected to?
E.g
Logs:
Jul 20 19:20:22 CST: %CONTROLLER-5-UPDOWN: Controller T1 0/0/1, changed state
to down (RAI detected)
Jul 20 19:20:38 CST: %CONTROLLER-5-UPDOWN: Controller T1 0/0/1, changed state
to up
Jul 20 19:20:38 CST: %ISDN-6-LAYER2UP: Layer 2 for Interface Se0/0/1:23, TEI 0
changed to up
Jul 20 19:20:40 CST: %LINK-3-UPDOWN: Interface Serial0/0/1:23, changed state to
up
Jul 20 19:37:28 CST: %CONTROLLER-5-UPDOWN: Controller T1 0/0/1, changed state
to down (AIS detected)
Sh int desc:
Se0/0/0:0
up
(DLCI 400 / EPVC 778)
Se0/0/0:0.1
up
(DLCI 400 / EPVC 778)
up
up
sh controller t1 0/0/0:
T1 0/0/0 is up.
Applique type is Channelized T1
Cablelength is long 0db
Description: AT&T MPLS T1 Circuit ID: DHEC.445470 (Data)
sh controller t1 0/0/1:
T1 0/0/1 is up.
Applique type is Channelized T1
Cablelength is long 0db
Description: AT&T T1 Pri Circuit ID: DHEC.104668.100 (Voice)
So in this case the T1 0/0/1 link flapped and its used for VOICE. We get a ticket
opened with AT&T under the ID: DHEC.104668.100.
How sure are you there were packets loss and performance degraded?
After entering sh controller t10/0/1 go all the way down.. youll see as below which
proves that circuit is bad.
Total Data (last 24 hours)
4 Line Code Violations, 5939 Path Code Violations,
224 Slip Secs, 2 Fr Loss Secs, 3 Line Err Secs, 44 Degraded Mins,
395 Errored Secs, 166 Bursty Err Secs, 7 Severely Err Secs, 60 Unavail Secs
----------------------------------------------------------------------------------------------------
(Router(CSU))====SJ=======TELCO GEAR======TELCO
GEAR========BGP NEiGHBOR Router
--->between CSU and SJ is LOS
--->between SJ and TELCO is AIS
--->between telco and telco is RAI
*SJ is smart jack*
----------------------------------------------------------------------------------------------------
BGP
Always related to a WAN link as its going to another router in the cloud.
Sh ip bgp summary:
Neighbor
V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.7.252.194 4 13979 50988 39516 52533 0 0 3w4d
2514
Meaning the bgp link has been up for 3 weeks and 4 days.
How to differentiate which bgp resides at which link?
Look at the IP in the bgp the link it uses will be the next hop IP address in the sh ip
int brief
Serial0/0/0:0.1
10.7.252.193 YES NVRAM up
up >>>>>
AT&T MPLS PVC Ckt ID: DHEC.455470 (DLCI 400 / EPVC 778)
Useful command:
sh ip bgp sum
sh log
sh clock to make sure the log time looks correct.
noticed that configuration was changed
looked for changes. None found.
looked for a site diagram.
none in the MDL.
looked for config changes in HP NA, but only one backup.
The 2 neighbors that are up are on Vzb MPLS Network.
---------------------------------------------------------------------------------------------------OOB : Out of Bound
When you get access, open Procomm. Go to top menu "Data" and select "Modem
Comand Mode"
Type in the window if it responds with OK. ats37=9 (sets modem to 9.6k)
atdt8,
To dial type atdt8,1 (phone number) example atdt8,1602 272-0802
Enter the phone number. Use 8 for long distance from Plano. Note: if the number
doesnt include the long distance access number, please include it. That would be a
1 for US and Canada and 011 for other countries.
----------------------------------------------------------------------------------------------------
International Paper
- IP - Only open tickets in US
- IP HD will open tickets with Orange on all International routers for IP (monitored
device)
-Just ignore those alarms.
-------------------------------------------------------------------Tower Perrin
Ignore tp alarms for crypto tunnels.
Note: TP (Towers Perin) crypto alarms are bogus, we do not manage those alarms,
just close them
Jpmntarcvtc001 is antoerh bogus alarm
-------------------------------------------------------------------CE router circuits that are attached to PE routers are handled by the Transport
group.
Never open a ticket on a CE device without asking if Transport is working it already.
####Event Update####
Spoke to ATT, advised NI can be looped but not CSU. This suggest a power outage.
Spoke to IP HD, they confirmed that there is power on site
Note: Never trust what they say. All they do is dial the modem and see if it rings.
Line protocol on ,Interface VirtualLine protocol on Interface VirtualLine protocol on Interface VirtualLine protocol on Interface Virtual-
When SE goes down, BRI ( ISDN backup ) come up. if it bounce for3 times, ticket
needs to be opened
---------------------------------------------------------------------------------------------------DUAL-5-NBRCHANGE: IP-EIGRP 132: Neighbor 10.3.246.89 (Dialer0) is down
usiwbrpd2303#sh int dialer 0
Dialer0 is up (spoofing), line protocol is up (spoofing) ack
Hardware is Unknown
Internet address is 10.3.246.1/24
MTU 1500 bytes, BW 128 Kbit, DLY 21000 usec,
reliability 255/255, txload 1/255, rxload 1/255
++++++++++++++++++++++++++++++++++++++++++++++
CRYPTO-5-SESSION_STATUS: Crypto tunnel is DOWN. Peer 192.85.7.12:500 Id:
192.85.7.12
MER015#sh crypto isakmp sa
dst
src
state
152.161.97.78 192.85.7.12
conn-id slot
QM_IDLE
++++++++++++++++++++++++++++++++++++++++++++++
CONTROLLER-5-UPDOWN: Controller E1 0/0 changed state to down (RAI detected)
RAI = remote alarm indication
--> occurs between the telco devices in the link path
reic-paris-477609#sh Controller E1 0/0
E1 0/0 is up.
Applique type is Channelized E1 - balanced
No alarms detected.
alarm-trigger is not set
Version info Firmware: 20040408, FPGA: 11
Framing is NO-CRC4, Line Code is HDB3, Clock Source is Line.
CRC Threshold is 320. Reported from firmware is 320.
Total Data (last 24 hours)
70 Line Code Violations, 4 Path Code Violations,
0 Slip Secs, 0 Fr Loss Secs, 4 Line Err Secs, 0 Degraded Mins, ack!!
4 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
----------------------------------------------------------------------------------------------------Eg:
Aug 14 18:50:19 CST: %SYSTEM_CONTROLLER-SP-3-ERROR: Error condition
detected: TM_NPP_PARITY_ERROR
Aug 14 18:50:19 CST: %SYSTEM_CONTROLLER-SP-3-EXCESSIVE_RESET: System
Controller is getting reset so frequently
Problem:
The switch reports this error message:
%SYSTEM_CONTROLLER-SP-3-ERROR: Error condition detected:
TM_NPP_PARITY_ERROR
This example shows the console output that is displayed when this problem occurs:
Feb 23 21:55:00: %SYSTEM_CONTROLLER-SP-3-ERROR: Error condition detected:
TM_NPP_PARITY_ERROR
Feb 23 22:51:32: %SYSTEM_CONTROLLER-SP-3-ERROR: Error condition detected:
TM_NPP_PARITY_ERROR
Feb 23 23:59:01: %SYSTEM_CONTROLLER-SP-3-ERROR: Error condition detected:
TM_NPP_PARITY_ERROR
Description:
The most common errors from the Mistral ASIC on the MSFC are
TM_DATA_PARITY_ERROR, SYSDRAM_PARITY_ERROR, SYSAD_PARITY_ERROR, and
TM_NPP_PARITY_ERROR. Possible causes of these parity errors are random static
discharge or other external factors. This error message indicates that there was a
parity error. Processor Memory Parity Errors (PMPEs) are are broken down into two
types: single event upset (SEU) and repeated errors.
These single bit errors occur when a bit in a data word changes unexpectedly due to
external events (which causes, for example, a zero to spontaneously change to a