Network Resiliency

Introduction
Ethernet has been a large success due to it being
standards-based, low-cost due to widespread deployment,
and because it is the most exible Layer-2 technology.
Ethernet supports countless numbers of topologies such as
stars, rings, and ladders.
While this exibility makes it ideal for a dynamic
environment, it also can create situations where additional
care is needed. When an Ethernet device receives
trafc with a destination Media Access
Control (MAC) address that is unknown

Ethernet
Topology Examples

to the device, the trafc will be

broadcast to all the devices
neighbors. This mechanism
allows the intended destination
device to receive the trafc
and reply to the source device.
On the reply, the device
sending back trafc will be
able to locate that destination
and create a forwarding entry
for it. The next time that trafc is
encountered with that destination
MAC address, the device can forward it
directly to the correct port without having to
broadcast the frame to all egress ports.

March 2006

Network Resiliency 1

Network Resiliency

Broadcast Storms
This broadcast mechanism is very effective. However, it can become an
issue in network topologies that include physical loops.
In the diagram below, when device 1 receives packets addressed
to an unknown destination, it will broadcast that packet on link A in an
effort to learn which of its ports the destination device is connected to.
When device 2 receives the packet, it will also attempt to learn which of its
ports the destination device is connected to. In doing so it will broadcast
the packet on link B. This pattern continues such that the packet is
broadcast on links C, D and E. Device 1 then receives the original packet
once again on link E as a result of device 5 broadcasting the packet.
Device 1 then broadcasts the packet again, attempting to learn which
port the destination device is connected to. For a network with a meshed
topology, this will result in the creation of thousands of packets, or what
is known as a broadcast storm. Broadcast storms result in diminished
link capacity for application trafc, and in severe cases can
render a network completely inoperable. Spanning Tree
Protocol (STP) was created to prevent broadcast

Broadcast Storm

storms.

1
A

B
3

4
C

2 Network Resiliency

March 2006

Spanning Tree Protocol

STP was dened by the Institute of Electrical and Electronics Engineers
(IEEE) standard 802.1D. STP prevents broadcast storms by automatically
and logically eliminating loops in an Ethernet network.
In the gure below, the network contains several loops, such as the loop
formed by links A-B-C and links B-D-E. To eliminate loops, STP uses an
algorithm described in the following paragraphs.
First, STP will select a root bridge. The root bridge is the logical center
of the network. Root bridge selection is accomplished by exchanging
messages called Bridge Protocol Data Units (BPDU) between the bridges,
or devices, that make up the network. Each BPDU contains the originating
device bridge identier. The bridge that has the lowest bridge identier
becomes the root bridge. Once the root bridge is selected, preventing
loops requires that one, and only one, bridge may forward trafc from the
direction of the root onto any given link.
STP determines a designated bridge and designated port
for each destination bridge. The designated bridge
is the only bridge that is allowed to forward

Network with Physical Loops

trafc from the root bridge to a particular

destination bridge. The designated port
is the port used to forward trafc away

from the root to the destination bridge.

The process of determining the

designated bridge and designated

port is accomplished by exchanging
BPDUs. The BPDUs contain path

costs, which are user congurable

weighting that enable prioritization of

one network path over another. The port
associated with the link that has the lowest
path cost becomes the designated port.

March 2006

Network Resiliency 3

Network Resiliency
Designated Bridge
and Port Selection
1

In the example above, all the physical links have a same cost of 10.
Possible paths to forward trafc from device 1 to device 4 are C, A-B,
and A-D-E. In this example path C would have a cost of 10, path A-B
would have a cost of 20, and path A-D-E would have a cost
of 30. Path C has the lowest cost, and therefore the

STP Enabled Network

port associated with that link will become the

designated port.
Once the root bridge and designated
bridges have been determined, STP

breaks the loops by articially disabling

links. Disabling a link is done by

putting the ports at both ends of the

link into a blocking state. A port that is

xE

in a blocking state will not transmit or

RSTP eliminated
physical loops

receive any trafc expect BPDUs. All

ports, except designated ports and root
ports, are put into a blocking state.
The bridges along the designated links

become the Spanning Tree.

4 Network Resiliency

March 2006

Rapid Spanning Tree Protocol

Once the Spanning Tree has been established, STP monitors the network
for failures and topology changes. Events that can cause the topology to
change include bridge failure, link failure, new links, new bridges, or new
bridge congurations. As these events occur, STP will automatically adapt
the tree to the new topology.
If link D fails, as shown in the gure below, bridge 4 becomes the
designated bridge for forwarding trafc to bridge 3. This topology change
occurs by going through the entire Spanning Tree Protocol algorithm after
the link failure. This process is called reconvergence.
STP was designed to support Enterprise networks where the recovery
from a failure within several minutes is acceptable. However, carrier-class
service delivery and guarantees require reconvergence times of less than
200 milliseconds.
To improve the time necessary to reconverge, the IEEE dened a
faster version of STP, 802.1w, called Rapid Spanning Tree
Protocol (RSTP). The goal of RSTP was to make the

Topology Change

reconvergence time close to one second, and

possibly less.

This performance improvement was

made possible by three categories of

Faster detection of a

topology change
n

important enhancements:

Simplied state machines.

STP denes ve port states,

where RSTP reduces that number

to a more efcient

If failure occurs,
RSTP will modify
the topology

set of three states

n

Simplied negotiation process

between the bridges

March 2006

Network Resiliency 5

Network Resiliency

LightningEdge Enhancements
World Wide Packets LightningEdge solution supports the delivery of time
critical services such as voice and video. In order to maintain acceptable
service levels during topology changes, reconvergence must occur within
200 milliseconds or less, much more rapidly than what RSTP alone was
designed to support.
The LightningEdge solution has been optimized to enable RSTP reconvergence
well below 200 milliseconds often below 50 milliseconds. These enhancements
allow Ethernet service delivery networks based on LightningEdge products
to support critical, time-sensitive applications with the same service level
agreements and guarantees of SONET/SDH optical rings.
These enhancements can be categorized using the same classication
as the RSTP enhancements dened in the IEEE 802.1w (now part of
802.1D-2004) standard.
They include:
n

More efcient state machines

n

More efcient negotiation process

The LightningEdge enhancements are

The

LightningEdge

fully compatible and interoperable with

IEEE STP and RSTP, and will operate

Solution

seamlessly in a mixed vendor network

deployment without any modication

Enables

to the BPDUs. However, the overall

Reconvergence

reconvergence performance of the

network will be signicantly enhanced

Times

with more LightningEdge devices.

as low as

This is due, in part, to our hardware

14 Milliseconds.

and software-based optimizations

for detecting link faliure conditions, as
well as optimizations of the protocol itself.
World Wide Packets will continue to enhance
our RSTP performance to achieve even faster

reconvergence times.

6 Network Resiliency

March 2006

LightningEdge RSTP Performance

World Wide Packets measures the LightningEdge RSTP performance
using a ring topology. Several parameters are changed between tests to
maximize the test coverage. Those parameters include:
n

Ring diameter: number of devices present in the ring.

MAC table size: number of MAC addresses present in the forwarding

tables during the tests.

Cause of the topology change: link failure, bridge failure, and new root
bridge.

In the test conguration, bridge 1 is the root bridge. Trafc is sent from
bridge 7 to bridge 2. Trafc will ow from bridge 7 to bridge 2 via bridge
1. To test reconvergence times, the link between bridge 1 and bridge 7
is disabled. After the topology change the trafc will ow via bridge 6, 5,
4 and 3. The number of frames dropped during the failure event is used
to determine the total time required for the trafc to reroute. World Wide
Packets denes this as the failover time.
Once failover is accomplished, the cause of the
topology change is removed, and trafc is restored

over the original path. World Wide Packets

denes the time required to reroute the

trafc as the restore time.

4
1
Primary
Path

March 2006

Failover
Path

Network Resiliency 7

Network Resiliency
LightningEdge Performance MAC Size
(7-node ring)

60

Time (ms)

50
40
30
20
10
0

1,000

2,000

4,000

8,000

16,000

Number of MAC Addresses

Fallover Time

Restore Time

The number of MAC addresses present in the forwarding table is a factor

in the RSTP performance. As the topology change occurs, the forwarding
tables need to be updated in order to forward the trafc onto the failover
path. The more MAC addresses that need to be updated, the longer the
reconvergence will take.

LightningEdge Performance Ring Size

(2,000 MAC Addresses)

Time (ms)

60
50
40
30
20
10
0
3

11

13

15

Ring Diameter (devices)

Fallover Time

Restore Time

The ring diameter is another factor in the RSTP performance. The more
devices that are present in the network, the more BPDUs have to be
exchanged, and the longer the negotiation process will take. The important
point here is that the failover time increases in a linear and predictable
fashion as the ring diameter increases.

8 Network Resiliency

March 2006

LightningEdge Performance Topology Change

Time (ms)

(7-node ring, 1,000 MAC Addresses)

90
80
70
60
50
40
30
20
10
0
New Root Bridge

Link Failure

Root Bridge Failure

Failure Cause
Fallover Time

Restore Time

The cause of the topology change is also a factor of RSTP performance.

A new root bridge will result in the least negotiation, while a root bridge
failure will result in the most negotiation.

Spanning Tree Domains

RSTP is a global protocol. It runs across a bridged Ethernet network. As the
access network grows, it may be desirable to partition the reach of
RSTP to certain areas of the network.
One example of this is illustrated by the gure
below, where the provider network is carried

Spanning Tree Domains

on a main ring to provide services to

edge rings that are connected to the
Edge
Ring

core provider ring. If a topology

change occurs on an edge ring, the

Main Ring

reconvergence event could impact the

main provider ring, and cause trafc

problems across the entire network.

The LightningEdge solution enables

Edge Ring

a network operator to dene areas of

BPDUs

the networks as Spanning Tree Domains.

Trafc within and between those domains
ows without any restriction. However, RSTP
BPDUs are only forwarded between bridges of the
same Spanning Tree Domain. This prevents a link failure
on one ring from causing trafc problems on other rings.

March 2006

Network Resiliency 9

Network Resiliency
The LightningEdge solution also offers Multi-Device Spanning Tree
Domains, a mechanism that enables an edge ring to be connected to two
bridges of the main ring for increased reliability and redundancy.

Multi-Device Spanning Tree Domain

Main Ring

Edge Ring

RSTP on Link
Aggregation Groups
As with all LightningEdge RSTP features,
including Spanning Tree Domains, are
fully compliant with the IEEE 802.1D and

802.1w standards. The format of the

BPDUs exchanged between bridges is

not modied.

Dynamic RSTP Path Cost

The RSTP negotiation process relies
on path cost to dene designated
bridges and designated ports. The
network operator denes the path cost
for each physical port.
If link aggregation groups are present in the
network (logical link between two devices created
by bundling several physical links together in order to offer
link redundancy or increase link bandwidth), the cost of a link can vary. If a
physical link part of a link aggregation group fails, the amount of trafc that
that logical link can carry decreases.

10 Network Resiliency

March 2006

In the previous gure, two link aggregation groups are present on a

network device. Link aggregation group A is forwarding, while link
aggregation group B is blocked by RSTP. If a failure occurs on A, the cost
of link A can become higher than the cost of link B.
The LightningEdge solution offers a unique mechanism to either statically
congure the cost of a link aggregation group, or dynamically relate the
link aggregation cost to its dynamic aggregation level, i.e. the number of
physical links aggregated.

RSTP Loopback Port Disable

The LightningEdge solution also offers a mechanism to prevent loops caused
by non-bridge network devices connected to a Lightning Edge device.
After sending a BPDU, if a LightningEdge device detects its own BPDU
coming back, it means that a physical loop exists on that part of the
network. The LightningEdge device will automatically disable RSTP on that
port and send a SNMP trap to alert the operator of the loop condition.
RSTP will periodically try to re-enable the port in order to
detect an eventual suppression of that loopback.

RSTP Loopback Port Disable

RSTP Management
All LightningEdge features, including RSTP,
can be congured via the Command
Line Interface (CLI), Simple Network
Management Protocol (SNMP), or from
the LightningEdge Network Supervisor
(LE-NS). Additionally, all LightningEdge
RSTP features can be enabled or
disabled independently.
The LightningEdge solution also
provides several SNMP traps that
facilitate troubleshooting of the network.
These traps are generated on events such as
the detection of incoming Per-VLAN Spanning
Tree (PVST) BPDUs, new root bridge, RSTP port state
changes, and RSTP loopback port disable.

March 2006

Network Resiliency 11

Network Resiliency

Conclusion
By offering unmatched RSTP performance, and a uniquely rich set of
RSTP feature enhancements, the LightningEdge solution raises RSTP
to carrier-class status, permitting network operators to leverage the
low cost and high-bandwidth of Ethernet to provision Service Level
Agreements with the reliability of SONET/SDH networks.

World Wide Packets LightningEdge

Solution
World Wide Packets LightningEdge Ethernet service delivery access
network solution, enables delivery of simultaneous telephony, business and
entertainment video, broadband data, and Internet access services, to any
combination of business and residential subscribers, using Ethernet over
ber and copper medium. By bringing together the limitless bandwidth
capabilities of a ber infrastructure with the affordability and proven
performance of Ethernet, these networks eliminate the constant bandwidth
challenges now facing network operators.
For more information on World Wide Packets, visit our Web site today, or
contact World Wide Packets at:
115 N. Sullivan Road
Spokane Valley, WA 99037
tel: +1.509.242.9000
fax: +1.509.242.9001
www.worldwidepackets.com
ISO 9001:2000 Registered
Access Brilliance, Ethernet to the Subscriber, ETTS, Fiber to the Subscriber, FTTS, LightningEdge, World
Wide Packets and the World Wide Packets logo are trademarks of World Wide Packets Incorporated. All
other trademarks are the property of their respective owners.

12 Network Resiliency

March 2006