Created by Dominique Verhulst Edition 2

Teleprotection over
packet networks
Legal notice
i
Copyright 2013, 2014 Alcatel-Lucent. All rights reserved
Alcatel, Lucent, Alcatel-Lucent and the Alcatel-Lucent logo are trademarks of Alcatel-Lucent.
The information presented is subject to change without notice. Alcatel-Lucent assumes no responsibility for inaccuracies contained herein.
All other trademarks are the property of their respective owners.
Editors notes
ii
Edition 1.2
This edition contains new information gathered since the rst edition. More
specically section 3 of chapter 2 has been updated with new user data and
illustrations.
New illustrations were added to chapter 4.
Edition 2.0
All animations have been reworked. More data was added from real current
di!erential and distance protection deployments over IP/MPLS networks.
Added references to standards.
A new section was added in chapter 2, covering bandwidth and latency
considerations to make when packetizing TDM data
The technical validation testing examples and practical use cases have been
grouped in a dedicated section in chapter 2. This allows more exibility to
add new examples in later editions.
A new chapter was added to clarify why IP/MPLS is chosen over MPLS-TP
A new chapter was added to explain the importance of synchronization
Acknowledgements
iii
This book was made possible through the help of numerous
colleagues, friends, customers and partners. At this point I would
like to thank all the people who have helped create this content,
provided inputs, feedback and inspired me to put this document
together.
In particular, I would like to extend my thanks to the following
people.
Dave Richards, Fai Lam, Benot Lridon, Bram De Valck, Dave
Sargent, Peter Merriman, Carl Rajsic, Colenso Van Wyk, Lieven
Levrau, Matthew Bocci, Hannu Ahola, Danny Knezevic of Alcatel-
Lucent Canada, France, Belgium, UK, South Africa, Finland,
Australia
Cory Struth, Clinton Struth of Falling Apple Solutions, Canada,
Patrick Colling, Michael Wener of Creos, Luxembourg,
Campbell Booth, Steven Blair, Federico Co!ele of Strathclyde
University, Scotland,
Andrej Grbing, Siemens, Germany
Fernando Castro, Joan Sans, Dimat-CG, Spain
Sincerely, Dominique Verhulst
Foreword
iv
Over the last few years Alcatel-Lucent has met many people from the power
utility and telecommunications industry and have come to the conclusion that
people who deal with some of the most mission critical applications in power
uti l i ti es, such as tel eprotecti on, have l i ttl e understandi ng of the
telecommunications infrastructure that assures the proper operation of their
applications. It is considered that the telecommunications system is always up
and has a guaranteed service performance. Likewise, the people with a
telecommunications background in general lack a good understanding of the
mission critical applications that power utilities need to operate in order to
assure 24/7 electrical power to our homes and businesses.
This book is meant to bridge this gap between the telecom and operational parts
of the power utilities organizations.
The hope is it can help the telecom people to appreciate the critical character of
the teleprotection applications and design better, future proof telecom networks.
Conversely, we also hope that this book can help instill condence with the
people who operate the teleprotection applications of the power utilities that IP/
MPLS can meet and exceed their requirements.
It can also be useful for service providers to understand the strict constraints of
the power utility applications in order to enable service providers to provide the
appropriate service SLAs that are required in the power utility market.
The Need to
Change
Since the War of Currents was won by the AC
camp in the late 19th century, electrical power
generation and distribution systems havent
changed much, other than that they have scaled
massi vel y. New cent ral power generat i on
technologies have been developed since then but
the continuing search for sustainable energy has
been driving the development of smaller scale
distributed power generation.
Power utility companies can no longer maintain the
status quo of how they deploy and manage their
power grid, there is an urgent need to change the
way they run their business, how they control and
protect the power grid. The grid has become a lot
more dynamic and this has a direct consequence
on the telecom network that is needed to control it.
Power utilities will need to embrace this change and
adopt new technologies to cope with the change.
v
Nicola Tesla is considered to be the father of the
AC power system. His patented inventions of the AC
induction motor and the transformer were licensed
by Westinghouse. This put Tesla in the AC camp in
the War of Currents
Chapter 1
Introduction
Power Utilities are managing one of the most critical
assets of our times: Electricity.
In order to ensure that the power grid is always on,
power uti l i ti es put systems i n pl ace that
continuously monitor the electrical infrastructure
and provide information about the state of the
power grid at key points in the network. These
systems are designed to help protect the power
grid against failures and avoid cascading of
problems through the grid. One of the key systems
that help power utilities protect their grid is called
the Teleprotection system. This interactive book will
provide you some insights into this application, how
it works and how it works in a telecommunications
network.
Power substation switchyard
Power Utility Assets - swipe to scroll through images
Why teleprotection?
Operating an electric grid requires the provision of safeguards. In particular, fault
detection and subsequent corrective action are extremely important.
Teleprotection is an essential requirement for operating and maintaining a reliable,
robust and safe electric grid. This functionality has always been required and its
Section 1
IN THIS SECTION
1. Why is teleprotection required?
2. What di!erent systems are there?.
3. Pilot based protection systems
4. Constraints of protection systems
5. Who are the players in teleprotection?
Introduction to teleprotection
7
Illustration 1.1 Power grid protection
Example of how power grids deal with failures - pinch open to enlarge, tap to start
animation
importance is magnied when Smart Grid deployments include
increasingly diverse sources of electric power that are combined
and channeled to increasingly diverse consumers of electric
power.
Teleprotection is a term that is often used to describe the systems
which will detect faults in the power grid and will activate circuit
breakers or reclosers to prevent faults from rippling through the
network or to restore power to a part of the grid after an outage.
Teleprotection systems can be classied into two main
groups:
1. Standalone protection systems
2. Pilot based protection systems
The rst group of teleprotection systems are referred to as
standalone teleprotection systems (also known as distance
protections).
Standalone teleprotection systems are monitoring the high
voltage line impedance and in case of a failure (and a resulting
change in impedance) determine where the fault is on the line and
make an autonomous local decision whether to trip the circuit
breaker or not.
The standalone protection system is shown in illustration 1.2
The second group of protection systems are referred to as pilot
based protection systems. The di!erence between this type of
system and the standalone teleprotection systems lies in the fact
that pilot based systems use a communications channel to allow
the teleprotection relays on either side of the high voltage line to
talk to each other and exchange information about what they
see on the power grid.
The illustration below shows how pilot based systems work.
8
High Voltage power line
Teleprotection Relay
Circuit breaker
Illustration 1.2 Standalone teleprotection system
Tap on the labels for more details, pinch to open to full screen.
Within the pilot-based Teleprotection systems group, we can also
make a distinction between two types of protection systems.
Pilot Based Protection Systems:
1. Distance Protection or Teleprotection Systems
2. Current Di!erential Protection Systems
The pilot based Teleprotection systems are the ones that are of
interest for this book since they rely on a communications
channel between adjacent nodes.Distance protection systems are
measuring the high voltage line impedance in order (see
illustration 1.4) to determine where the fault is, just like the
standalone system, however since they work in pairs to monitor
both sides of the line, pilot based Teleprotection systems can
make better and faster decisions with regards to tripping their
local circuit breaker. There is a possibility that the fault is further
upstream or downstream in the grid. Therefore, information from
an upstream or downstream neighbors Teleprotection relay can
be essential in making the right decision to trip the circuit breaker
as close as possible to the fault. Teleprotection systems send
commands on an event basis. They can tell their neighbor to
9
Tele- or Distance Protections work on the principle that they
measure the impedance of the high voltage line to determine the
location of a fault.
Illustration 1.4 Principle of teleprotection
Pilot communications line
Illustration 1.3 Pilot based teleprotection
block or to trip in case they see a fault. These protection
schemes are generally referred to as blocking and permissive
protection schemes. The animation in illustration 1.4 shows the
principles of these protection schemes.
Current Di!erential Protection systems work di!erently from the
Tele- or Distance Protection systems. They use a di!erent
mechanism to determine if there is a fault condition on the grid.
The principle of operation of Current Di!erential protection
systems is based on a basic rule of electrical systems: Kircho!s
point rule. This rule says that the sum of the currents owing into
a point is always identical to sum of the currents owing out of
the same point. If there is any di!erence, there is a fault.
Current Di !erenti al
relays continuously
send sample values to
e a c h o t h e r t o
compare the current
values that they see
(as in illustration 1.6).
The di!erential relays
rely on a xed latency
time to transmit and
recei ve t hei r dat a
because they need to
compare real -t i me
sample values and use a xed o!set to compensate for the
communications latency. Hence the need for low latency,
accurate timing and very low variation in the latency.
Constraints of Teleprotection systems
A failure such as a short circuit in a high voltage power system
can cause major damage that can lead to an avalanche e!ect on
the entire power system of a city, region or country. It is therefore
10
This animation shows the principles of operation of blocking and
permissive protection schemes.
Illustration 1.6 Distance protection schemes
Illustration 1.5
Kircho!s point rule
essential that failures are detected rapidly and that protection
systems can be engaged immediately after a failure is detected.
Todays teleprotection applications are typically developed using
a total fault clearing time of ve to seven electricity cycles. The
electricity cycle is at the rate of 50Hz (20 ms per cycle) in most
parts of the world and typically at 60Hz (16.66 ms per cycle) in
the Americas. As shown in illustration 1.8, the fault inception (1 in
gure) and fault resolution delay (4 in gure) take between one
and three-to-ve cycles, respectively. This leaves one cycle or
16.6 ms (20 ms in a 50Hz grid) for total end-to-end delay
comprised of teleprotection (TPR) equipment delay (2 in gure)
and telecom network delay (3 in gure). So if teleprotection relay
delay was found to be around 3 ms at each terminal, this leaves
approximately 10 ms for an acceptable total telecom network
delay. The 10 ms maximum latency for digital systems is dened
in the IEC 834-1 standard. Older teleprotection systems that are
11
This animation shows the di!erent steps in Teleprotection fault clearance
and the time associated with each step.
Illustration 1.8 Teleprotection fault clearing
Current di!erential relays continuously send and receive sample values
to compare currents, if they measure di!erent values, the relay is
tripped.
Illustration 1.7 Current di!erential protection scheme
using analog (voice band) communication interfaces can be
allowed up to 15, 20 and even 40 ms in some cases.
Latency is not the only constraint to which the teleprotection
application is subjected, the variation of the latency (even if it is
still within the overall 10 ms limit) is also a very important factor.
Latency variation is also often referred to as Jitter.
For teleprotection applications the Jitter tolerance is less than
1ms, typically 0.5 ms.
Pilot based teleprotection systems rely on a communications
channel between them in order to send and receive information.
This means that information is sent from both sides to the other
side of the protected line. Hence we have two communications
paths, one forward and one reverse path. For Teleprotection to
work properly it is important that the latency is the same for both
forward and reverse paths, the di!erence has to be less than 2ms
and needs to be constant (within the Jitter tolerance). This is
often referred to as path asymmetry, or the di!erence between
the forward and reverse path.
Another standard that denes delivery times for power substation
applications is IEEE 1646. Generally it denes latency for
protection applications between 8 and 16 ms.
Who are the players in the teleprotection market?
The vendors who are active in the teleprotection relay market
have been there for quite some time. Names such as ABB,
Alstom Grid, General Electric, Schneider Electric, SEL, Siemens
and ZIV-Dimat are often found in the Power Substations
environments.
12
Illustration 1.9 Time constraints of teleprotection
Section 2
Test your knowledge of teleprotection
13
Review 1.1 Teleprotection Principles
Check Answer
Question 1 of 5
What is the purpose of teleprotection?
A. To monitor the condition of the
power grid
B. To isolate faults in the power grid
C. To prevent damage to critical parts
of the power grid
D. All of the above
Chapter 2
Communication
Networks for
Teleprotection
This chapter will cover the communication
technologies that are used in power utility networks
and more specically the ones that are used for real
time applications such as teleprotection.
Before we start going into more detail about the communication networks which
are used for teleprotection applications, it is worth elaborating a little on the
communications interfaces used by teleprotection equipment.
Most older generation teleprotection equipment is using analog voice interfaces.
The teleprotection equipment basically sends audio tones across the
communications link which could be as basic as simple copper wires. There are
di!erent frequencies that are used for the guard tone (to signal that the
teleprotection relay is operational) and for the commands channels. The
frequencies are all within the audible spectrum from 1100Hz up to 4000Hz. The
frequencies and modulation schemes are dened by ITU-T standards. Commands
are typically sent at very low speed (200 baud or lower) using frequency
modulation.
The physical interface used by analog teleprotection equipment is most commonly
the 4 wire E&M interface. This interface uses 6 wires; 2 wires for transmit, 2 for
receive and 2 for signaling.
More recent teleprotection equipment is using digital interfaces. Over the past 20
years or more, there has been an evolution of these digital interfaces; the most
common ones found on teleprotection relays are X.21, G.703, IEEE C37.94 and
most recently Ethernet interfaces.
Section 1
IN THIS SECTION
1. Analog interfaces
2. Digital interfaces
Interfaces of teleprotection equipment
15
Guard Tone
Channel
Illustration 2.1 Frequency spectrum of
analog teleprotection systems
Teleprotection relays rely on a stable, symmetric, constant delay telecom network.
Traditional telecom networking utilizes TDM transmission architectures based on
PDH/SONET/SDH to provide the communication channel between relays. The
circuit switched nature with xed frame lengths provided some guarantee of delay
limit, delay stability and transmission symmetry. Existing PDH/SONET/SDH
Section 2
IN THIS SECTION
1. Traditional network architectures
2. Limitations of traditional networks
3. Other factors which drive the need to
change
Legacy Networks
16
Illustration 2.2 Typical traditional network architecture and frame structure
The trouble with our times is that the future is not what it
used to be.
- Paul Valry - French Poet and critic (1871-1945)
networks have proven to be well suited to the task of supporting
current di!erential protection, which has demanding performance
imposed upon the communications networks.
Over the past few years, there has been a growing demand for
more Ethernet and IP based services to be provided for power
substations. Engineers are expecting to be able to connect to
their corporate intranet from within the substation. Analog
telephones are being replaced by Voice over IP models, CCTV
cameras are moving to IP and multicast based solutions.
But perhaps the most important driver towards Ethernet and IP in
the substation is the deployment of IEC 61850 based substation
automation systems. These require Ethernet connectivity and
hence we see more Ethernet switches and routers being
deployed in the power utilitys communications networks.
While recent SONET/SDH equipment does provide Ethernet
connectivity, they are ill equipped to deal with the scale and
complexity of o!ering Ethernet and IP services for large power
utility networks.
The problem with non-MPLS based IP routing and Ethernet
switching technologies however is that they arent well suited to
deal with real time data from applications such as SCADA and
teleprotection, so other solutions need to be considered.
Besi des the l i mi tati ons of the exi sti ng power uti l i ty
communications networks, there are a number of other business
related factors that are driving the need to change.
The drive for Power Utilities to transform their telecom and
corporate Informati on and Communi cati on Technol ogy
department (ICT) into an integrated operations and ICT
organization is the direct result of the need to reduce costs,
maintain or increase performance while making the transition to a
smart gird enabled company.
17
This animation shows how Power Utility companies are transforming
their operational and ICT departments into a converged organization in
multiple steps.
Illustration 2.3 Power Utility telecom transformation
This organizational transformation poses some signicant
challenges for the network infrastructure because the operational
telecoms applications are mission critical and have fundamentally
di!erent requirements in terms of bandwidth, end-to-end-delay
and jitter compared to the corporate ICT applications. The latter
have been deployed on Ethernet and IP networks for a decade or
more while the rst have relied on more conventional TDM
technologies for several decades. Bringing those two di!erent
worl ds together onto a common tel ecommuni cati ons
infrastructure is by no means a trivial task. However, the benets
are signicant and the transformation must happen.
Why is this happening and why now?
There are several key drivers forcing this transformation and
making it very relevant and timely.
The Smart Grid. There is a big push from governments
worldwide towards smart grid capabilities which have a
signicant impact on how power utilities have to manage demand
and supply and be able to more rapidly adjust to both. This
results in more sensors and actuators to be deployed producing
more data to be collected and transmitted in real time. At the
same time, applications using that data are communicating with
various systems in the network.
Security Issues. Coupled with government demand for smart
grid capabilities is the increase in security directives. These are a
result of the increased terrorist and intrusion threats to power
infrastructures. These directives are having a direct impact on the
network.
18
Technology Lifecycle Issues. Most of the TSO and DSO
communications networks have been using TDM technologies for
the past 20 years or more. Government regulations such as RoHS
and major shifts in technology in the service provider and
enterprise markets are making it very di"cult for the vendors of
TDM technologies to maintain their investment in the
development and production of this technology. Most service
providers and enterprise customers have evolved their
communications networks to other technologies such as Ethernet
and IP which resulted in a steady
decrease of demand for TDM products
over the past decade. Consequently,
there is considerably less investment in
the development and support of TDM
technologies. There are fewer vendors on
the market with native TDM products
while vendor interoperability and end-to-
end management is often a big issue.
Along with this aging of technology,
maintenance costs are increasing and
nding skilled sta! to operate and
maintain these legacy technologies and
equipment is proving to be very di"cult.
Leased Line Service Migration. As
al ready menti oned earl i er, servi ce
providers are changing their network
infrastructure to packet based technologies in order to reduce
their costs, deal with increased bandwidth demand, support new
applications and o!er new revenue generating services. Service
providers therefore are greatly reducing traditional leased line
services to the point where they are no longer o!ering TDM
based leased line services in many areas. Power utilities often rely
on leased line services either as backup to their own network or
to reach remote substations. As a consequence of the service
providers decision to no longer o!er these services they are
forcing packet services on power utilities.
Own Fiber Optic. Power Utilities
have deployed ber optic cable
(for instance through new OPGW
Overhead Powerline Ground
Wire) through much of their
i nfrastructure whi ch provi des
considerably more available bandwidth
to enable the transition to converged, packed-
based networks.
Evolution Of Microwave Technology. Many
power utilities have deployed microwave
communi cat i ons syst ems t o connect
substations to the communications network in
areas where ber or leased line services were
not available or because microwave was the
19
The Newbridge MainStreet 3600 was a TDM
multiplexer that has been sold from 1987 till
2010. It was the worlds rst software managed
TDM multiplexer
Illustration 2.4 MainStreet 3600
most cost-e!ective solution. As
well, service providers have been
upgrading their mobile backhaul
networks from TDM to packet
based technologies to keep up
with the demand for bandwidth
and support of IP. This is driving
the development of high capacity
and quality of service (QoS)
b a s e d m i c r o w a v e
communications systems, well
suited to the utility and Smart
Grid needs.
New Technologies Mature.
Viable packet based alternatives to TDM and SDH/SONET
technologies are mature now and have been deployed widely.
Some vendors have implemented new standards on some of their
products that support mission critical power utility applications
such as Teleprotection, SCADA, Telemetering and Telecontrol to
work awlessly on an IP/MPLS based packet network. However,
all IP/MPLS implementations are not equal so power utilities will
have to ensure their needs are met when selecting vendors.
New Standards Evolve. Standards bodies such as IEC are
working on standards for packet based intra- and inter-substation
automation and telecommunication (IEC61850-90-1).
Cost Reduction. Service providers have
moved away from SDH and PDH technologies
because Ethernet has become a lot more cost
e!ective. As a consequence, they are o!ering
higher speed packet based services as a
replacement for leased line services at much
more attractive price points. Power utilities can
take advantage of this and will save costs on leased lines and
also on their own network equipment by moving to more Ethernet
based solutions.
Government Funding. In order to ease the move towards Smart
Grids and the use of more green energy, governments are making
funds available for research, trials and full scale rollouts.
All of the above is resulting in an accelerated demand from Power
Utilities to help them understand the new technologies and
methods to transform their networks to modern, converged,
reliable and agile infrastructures.
20
Illustration 2.5 Alcatel-
Lucent 9500 Microwave
Packet Radio
With the advent of IP and Ethernet devices in the power systems, many utilities are
migrating their telecom networks to IP/MPLS in order to support next-generation
interfaces and to achieve a converged network infrastructure for existing, new as
well as smart grid applications. From a telecommunications requirement for
teleprotection, it is necessary to continue providing the connectivity between
relays with a high quality of service quantied by limits on delay, delay variation
and asymmetry. An IP/MPLS network can meet these as well as other key
requirements such as reliability, exibility, manageability and maintainability.
A major concern for utilities is whether the IP/MPLS network can meet the strict
latency requirements for protection signals between transmission substations.
The doubts over IP/MPLS usually concern the ability to guarantee low latency
service. First, IP/MPLS is sometimes and incorrectly perceived as connection-less
IP-technology that can provide data transport but only with a best-e!ort like
quality of service (QoS). This is the case for IP only however and in contrast, the
MPLS part of IP/MPLS makes the solution connection-oriented and capable of
multiple guaranteed quality of service (QoS) levels.
Another concern about IP/MPLS networks, as applied to teleprotection schemes,
is the notion that the statistical nature of packet networks will impact the
performance of teleprotection relays. This notion has been disproved through
extensive testing and implementation and is not applicable when the Utility
Section 3
IN THIS SECTION
1. Misconceptions about IP/MPLS
2. Underlying principles
3. Performance of Teleprotection over IP/
MPLS
4. Provisioning and Managing Teleprotection
services on an IP/MPLS network
Addressing the challenges with IP/MPLS
21
It is not the strongest of species that survives, not the
most intelligent, but the one most responsive to change
Charles Darwin English naturalist & geologist(1809-1882
operates a private network based on IP/MPLS and, as with a
traditional PDH/SONET/SDH network, suitable care is taken in the
engineering and provision of the network.
SONET/SDH networks can be provisioned to provide alternate
routes for mission critical tra"c such as that between
teleprotection relays. When operating correctly, the network
provides less than 50 ms switchover time. This 50 ms time is the
reference in terms of resiliency for any new telecom technology.
IP/MPLS network technology provides several di!erent fully
standardized resiliency mechanisms to provide a switchover time
of less than 50 ms. These include end-to-end alternate paths and
MPLS Fast ReRoute (FRR). MPLS Fast ReRoute provides many
pre-calculated alternate paths that can overcome any failure
scenario in less than 50 ms. On top of that Alcatel-Lucent
provides Non-Stop Service (NSS) capability to the applications
riding over the IP/MPLS network. More details about IP/MPLS
and the reasons why this is the best long term solution over for
instance MPLS-TP is provided in chapter 4.
The Alcatel-Lucent IP/MPLS network supports teleprotection with
many features:
# IP/MPLS networks utilize Label Switched Paths (LSP) to
ensure that all packets associated with a particular service, such
as teleprotection, follow the same path. This is often referred to
as strict paths. This ensures that the predetermined latency target
is always met.
# The packets associated with teleprotection communication
can be assigned a high priority to guarantee that the criticality of
teleprotection requirements are met and reduced packet delay
variation through the network assured.
# The Alcatel-Lucent IP/MPLS network supports many
synchronization options to ensure that the network is properly
synchronized. Since the IP/MPLS routers are synchronized, they
can provide a good reference clock to the relays that are
connected using serial interfaces by using the Network Clock to
generate the Service Clock. Next generation relays connected
using Ethernet can also be synchronized since the Alcatel-Lucent
22
The Alcatel-Lucent 7705 SAR-8 is a fully redundant and
environmentally hardened IP/MPLS router with eight modular slots.
Illustration 2.6 Alcatel-Lucent 7705 SAR, a
teleprotection capable IP/MPLS router
IP/MPLS routers support ITU-T Synchronous Ethernet (SyncE)
and IEEE 1588v2 Precision Time Protocol (PTP). See chapter 5 for
more details on synchronization.
# Alcatel-Lucent IP/MPLS routers natively support commonly
used teleprotection interfaces including E&M, RS-232, X.21, G703
and IEEE C37.94. It also supports Ethernet for next generation of
Ethernet based relays. To reduce latency, it is advantageous to
support a direct interface from
the teleprotection relay to the IP/
MPLS routers. This eliminates
the need for a channel bank and
the additional latency that is
added.
Underlying principles. An IP/
MPLS ne t wor k s uppor t s
traditional relays using Circuit
Emulation Service. The key
desi gn consi der at i on f or
supporting Teleprotection over a
packet network i s how to
minimize latency. For an IP/
MPLS network, the telecom
network latency for TDM tra"c
over I P/MPLS consi st s of
packetization delay, network
transport delay and jitter bu!er/depacketization delay.
Packetization delay relates to the process of transforming TDM
tra"c into packet data. For network delay, there is xed delay
based on physical link speed and distances involved and a
variable delay depends on the number of hops (nodes) in
between. Thereby each node adds transit equipment latency.
Jitter bu!er and de-packetization delay relates to the time
required for a data packet to
move out of the jitter bu!er and
to get de-packetized into a TDM
st ream connect i ng t o t he
teleprotection relays.
The majority of the telecom
network delay occurs at the
edge where low speeds are
present. Latency in the core
depends on number of nodes
traversed. The use of coloring
and strict paths can be used to
engineer the network to avoid
p a t h d i v e r g e n c e a n d
asymmet ri cal del ays. I n
addition, the IP/MPLS router or
a GPS clock can provide a real-
time clock pulse that can be
23
This animation shows how serial data is packetized and sent across an
IP/MPLS tunnel to the remote location where the jitter bu!er ensures a
smooth play out of the serial data
Illustration 2.7 Packetization delay
used for relays to time-stamp their data and thus ensure accurate
data comparison and processing.
Packetization delay is the delay imposed at ingress as the TDM
data is packetized. Smaller payload sizes with a higher number
of packets per second result in lower packetization delay and
lower one-way delay. Larger payload sizes with a lower number
of packets per second result in higher packetization delay and
higher one-way delay. The packet payload size is congurable.
Network transport delay in the core depends on the number of
nodes, distance and communications medium, but results mainly
from transmission delays. With IP/MPLS tra"c engineering, a
service such as teleprotection follows a pre-determined path
through the network that meets the strict latency requirement.
On play out, a Circuit Emulation Service uses a jitter bu!er to
ensure that received packets are tolerant to packet delay variation
(PDV). This ensures the successful de-packetization of the
payload back into the TDM stream needed for communication
with the teleprotection relay. The smaller the jitter bu!er, the less
delay is imposed. However, the jitter bu!er needs to be set at a
large enough value to ensure that jitter cannot cause a
communications failure on the teleprotection relays. The selection
of jitter bu!er size must take into account the size of the TDM-
encapsulated packets. Larger payloads will require larger jitter
bu!er sizes. A properly congured jitter bu!er provides
continuous play-out, thereby avoiding discards due to overruns
and underruns.
Another key element to assure the correct functioning of TDM
appl i cati ons over a packet network i s the ti mi ng or
synchronization. Over the past few years, several techniques
have been developed to provide high precision timing over packet
networks. The most important ones to consider for the use in
teleprotection applications are Synchronous Ethernet and IEEE
1588v2. Both technologies scale well in large networks and
provide timing precision that is appropriate for the teleprotection
24
IP/MPLS enables simplication of the network and supports all critical services
in a deterministic way.
Illustration 2.8 IP/MPLS network for power utilities
systems. However Synchronous Ethernet does have the
advantage that it is a layer 1 based synchronization solution and
therefore is totally immune to packet delay variation and it
interworks well with SONET/SDH network synchronization. This
is an element that can be important in migration scenarios.
Alcatel-Lucent IP/MPLS routers support both Synchronous
Ethernet and IEEE 1588v2, allowing complete exibility in the
synchronization design of the network.
Next generation relays now utilize Ethernet interfaces. Point-to-
point relay communication can be supported with an IP/MPLS
network using Ethernet Virtual Leased Line (VLL) service and
multipoint communication such as IEC 61850 GOOSE messaging
can be supported using Virtual Private LAN Service (VPLS).
Managing an IP/MPLS network for power utility applications.
IP/MPLS is proving to be capable of handling the teleprotection
tra"c. However one of the key concerns of power utilities to
embrace this technology is the perception that managing IP/
MPLS networks, provisioning services and managing alarms is
very complex.
Alcatel-Lucent has spent the past ten years developing a suite of
network and service management products which leverage the
many years of experience in managing PDH, SDH and ATM
networks in order to simplify the task of managing IP/MPLS
networks. One of the main components of this suite of
management products is Alcatel-Lucent 5620 Service Aware
Manager (SAM). This product leverages more than 25 years of
know how in network and service management software
development. It is building onto the experience of the 5620NM
products that were devel oped to manage the former
MainStreet3600 TDM network products.
5620 Service Aware Manager is a client-server based
management system that supports up to 50 concurrent clients
which can be set up according to specic rules related to span of
control and scope of command.
25
5620 Service Aware manager, manages end-to-end services, network
elements, all parts of the network layers, synchronization, SLAs and
much more.
Illustration 2.9 5620 Service Aware Manager
5620 SAM can be congured in an active/standby server
conguration. The switchover happens automatically in case the
active server fails, or it can be triggered from the command
interface.
5620 SAM allows thousands of nodes to be managed from the
server. It allows the conguration of the nodes, creation of the
point-to-point, point-to-multipoint and meshed services from an
easy to use graphical interface. It monitors services against
specic service level agreements (SLA). SAM provides the
visualizing of all the paths and the services in the network in the
network and can raise an alarm in case a path is re-routed and
fall outside the limits set by the service SLA. Furthermore, 5620
SAM has a northbound interface, based on XML that allows the
integration with OSS systems from IBM, HP and others. It also
o!ers the ability to create web based service portals in order to
further simplify the day-to-day operation of the network, creating
reports etc. To ensure secure communication between 5620 SAM
and the OSS system, the XML interface supports encryption.
The Alcatel-Lucent 5620 SAM provides:
# Easy-to-use graphical forms for point-and-click element,
network and service conguration
# Wizards to guide administrators step-by-step through complex
tasks
# Advanced scripting, templating and rules-based conguration,
allowing customization of the Alcatel-Lucent 5620 SAM for
specic network or service requirements. This customization
also allows non-expert resources to handle more complex tasks
and eliminates repetitive data-entry activities
It is not only important to have the tools to manage the elements,
services, protocols, it is equally important to be able to plan
ahead. For this Alcatel-Lucent o!ers di!erent options. First there
is the 5650 Control Plane Assurance Manager which is integrated
with the 5620 Service Aware Manager. It allows to perform some
o!-line simulations. This means that users can look at di!erent
what if scenarios such as, what if this link fails or what if this
node fails. Network modeling an capacity planning can be
integrated with third party applications such as these from Aria
Networks and Opnet, seamlessly integrated with 5620 SAM
through the open XML interface.
26
Illustration 2.10 5620 SAM screenshot samples
Further automation and zero-touch conguration, if required, can
be provided by deploying a service portal such as the Service
Portal Express for Utilities.
The Alcatel-Lucent Service Portal Express for Utilities is a
lightweight, web-based application tightly coupled with the
Alcatel-Lucent 5620 Service Aware Manager (SAM). It is purpose-
built for utility operators to simplify IP/MPLS network operations
and management, delivering utility-specic workows and
terminology to bridge the gap between network experts and
operators with less extensive training. For added security and
control, workows may also be routed to proper authorizations for
review and approval.
The Alcatel-Lucent Service Portal Express for Utilities enables
rapid, cost-e!ective deployment by providing a framework-
based architecture that is both modular and extensible.
Base modules are included to provide the following
capabilities:
# Provisioning
# Monitoring
# Troubleshooting
# Reporting
27
Illustration 2.11 Service Portal Express for Utilities is making
provisioning, monitoring and reporting extremely simple.
Packetization Principles
In the previous section we covered the underlying principles of the packetization
technique which is used to take data from a serial (i.e. X.21) or n x 64kb (i.e. E1, G.
703, C37.94) interface and send it across an IP/MPLS network. In most cases
there is an optical ber infrastructure available that o!ers plenty of bandwidth, so
there is no immediate need to focus on the bandwidth consumption impact of
packetization. However, there are a number of cases where there is no ber optic
network available as is the case when using microwave radio links or copper lines
with xDSL modems.
In those cases, it is important to understand the impact of the parameters, used to
congure the packetization system, on the bandwidth requirement that results
from these parameter settings. Therefore it is worth looking at the principles of
packetization in a little more detail.
Packetization is the process that occurs when legacy interface data (serial data or
data coming from DS0 channels in a T1/E1) is presented to a routers ingress port.
The router needs to take a series of bytes and put them into packets
(=encapsulation) at a certain constant rate which then get routed to their
destination. This is done according to a standard mechanism as dened in the
CESoPSN standard (RFC 5086). An alternative mechanism to transport TDM over
Section 4
IN THIS SECTION
1. Packetization principles
2. Impact of jitter bu!er and payload size
conguration on bandwidth requirement
3. Validation testing
4. Examples of implementations
Bandwidth & Latency considerations
28
a packet network is Structure Agnostic Transport over Packet
(SAToP). This method is not taking the DS0 channels into
consideration and basically transports the complete E1 or T1
transparently.
The Jitter Bu!er assures a smooth play-out of the serial data on
the egress port and compensates for packet delay variations that
may occur during transit of the packet through the network and
intermediate network nodes.
When a CESoPSN circuit is provisioned on IP/MPLS routers we
will need to dene the payload size (in bytes) at the ingress port
and the Jitter bu!er depth (in ms) at the egress port.
The Impact Of Payload And Jitter Bu!er
The conguration of both parameters will have an immediate
impact on the number of packets that will be generated and the
bandwidth consumed on the network ports.
Lets consider an example whereby we packetize data from an E1
interface.
The E1 frame structure consists of 32 bytes with each byte
representing a di!erent time slot of the E1 frame: TimeSlot 0 to
TimeSlot 31. With the E1 interface speed being 2.048Mb/s, each
E1 frame takes 125 microseconds. The structure of an E1 frame
is illustrated in illustration 2.14. The E1 structure consists of
cycles of 16 frames which ensure CRC4 error checking. The time
it takes to complete a full E1 cycle (16 frames) becomes 16 x
0,125 ms = 2 ms.
When packetizing data such as that from an E1 or T1 circuit,
there are a few key parameters that will have an important
inuence on the resulting end-to-end latency and the bandwidth
requirements. These parameters are the payload size (PS), this is
the amount of data (in bytes) we take from the E1 or T1 circuit (or
other serial data) to put in each packet. For an E1/T1 circuit this
29
An E1 frame consists of 32 PCM channels or time slots , which means
they are 8 bits in length each. Timeslot 0 is used for framing and error
correction, timeslot 16 is used for signaling purposes of the voice
channels in timeslots 1-15 and 17-31.
Illustration 2.12 The E1 frame structure - tap to play
would be the number of time slots (N) multiplied by the number of
frames (F) we want to send per packet. For the purpose of
calculating the formula is:
PS = N x F is expressed in bytes
We know that our frame rate (FR) is one frame every 125
microseconds.
With this information we can calculate the packetization delay
(PD) by multiplying the frame rate with the number of frames we
want to send per packet.
PD = FR x F typically expressed in ms
Every telecom equipment such as a router will introduce a xed
delay, this will be in the range of tens or hundreds of
microseconds depending on the architecture of the product.
This xed delay (FD) needs to be added to the calculation of our
total delay.
The next important parameter is the jitter bu!er size. The concept
of the jitter bu!er has been explained in section 3. The Jitter
Bu!er is needed to assure a smooth ow of data and minimize
packet delay variation. The jitter bu!er size (JB) is a parameter
that is congurable, typically in increments of 1ms. When a router
starts to play-out the data it receives at 50% of the jitter bu!er,
the resulting jitter bu!er delay (JBD) is half of the congured JB
value.
The total delay (TD) therefore becomes the sum of the
packetization delay, the xed delay and the jitter bu!er delay.
TD = PD + FD + JBD
A parameter which will have a crucial impact on the bandwidth
utilization is the packet rate (PR) in packets per second, this is the
result of the product of the number of E1/T1 frames we chose to
put in each packet (F) with the frame rate (FR) which is 0,125 ms
or 0,000125s for E1. The formula to calculate the packet rate is
PR = 1/(FR x F) in packets per second
To calculate the resulting bandwidth (BW) in bits per second we
need to add the protocol overhead (MPLS, Control Word,
Ethernet, ML-PPP,...), which is typically 42 bytes to the payload
size (PS) and multiply by the packet rate times eight (eight bits
per byte).
BW = (PS+42) x PR x 8 = bandwidth in bits per second
Lets consider the following example.
We are transporting one timeslot of an E1(N=1) and we will put 16
frames in each packet (F=16), therefore the payload size is
PS = N x F = 1 x 16 = 16 bytes
30
The packetization delay becomes
PD = FR x F = 0,125ms x 16 = 2 ms
Lets assume a xed delay of 0,3 ms and we congure a jitter
bu!er to one ms, then the total delay becomes
TD = PD + FD +JBD = 2 + 0,3 + 0,5 = 2,35 ms
The packet rate in this example would become
PR = 1/(FR x F) = 1/(0,000125 X 16) = 500 packets per second
And so the resulting bandwidth requirement would be
BW = (PS + 42) x PR x 8 = (16 + 42) x 500 x 8 = 232 kbits/s.
As can be seen from these calculations, careful considerations
must be made when putting TDM tra"c over packet networks.
When available bandwidth is constrained for instance over
copper lines or microwave links, one must pay attention to the
parameters used for the packetization of the data. It is important
to understand the boundaries of the application in terms of
maximum latency, apply a margin to it and work the numbers
back to nd the right balance in terms of end-to-end latency and
bandwidth requirements.
In networks where the links are built with ber optic connections,
bandwidth is less on an issue and therefore tuning the parameters
towards the lowest possible latency is not a problem.
The illustration below shows a sample graph of how the number
of E1 frames one choses to put into a single packet has an
impact on the bandwidth utilization.
31
Illustration 2.13 Sample table of packetization bandwidth
0
200
400
600
800
32 16 8 4
736
400
232
148
b
a
n
d
w
i
d
t
h

i
n

k
b
/
s
number of E1 frames per packet
Performance of teleprotection over IP/MPLS
Alcatel-Lucent engaged Iometrix, the networking industrys preeminent testing and
certication authority, to test and validate the ability of the Alcatel-Lucent IP/MPLS
based 7705 Service Aggregation Router (SAR) and 7750 Service Router (SR)
products for implementing an IP/MPLS network to support teleprotection. This
testing was done in collaboration with Toshiba who provided their GRL100 relay
equipment and engineering personnel who participated in the testing and veried
proper performance of the relays. The Toshiba equipment was available in both X.
21 interface and Ethernet interface versions, allowing the verication of support for
both traditional and next generation teleprotection relays.
Based on a comprehensive battery of tests, it was concluded that a network
comprised of Alcatel-Lucent IP/MPLS routers will comply with all the requirements
of teleprotection with substantial margin. The IP/MPLS network performed well
within the requirements of the teleprotection application that has, to this point,
only been supported by circuit switched (TDM) network (e.g. based on SONET/
SDH) devices. The Alcatel-Lucent routers support legacy and next-generation
device interfaces such as Ethernet and consequently can support both existing
teleprotection devices as well as those that will be deployed in the future. This
capability is crucial for smooth migrations of utility networks.
Section 5
IN THIS SECTION
1. Performance and validation of
Teleprotection over IP/MPLS
2. Examples of implementations
Performance Validation and Use Cases
32
Illustration 2.14
Toshiba GRL-100
If I have seen further it is because I could stand on the
shoulders of giants
Isaac Newton English scientist (1643-1727)
In addition to the Iometrix validation, Alcatel-Lucent and third
party laboratories have also conducted testing, sponsored by
utilities that evaluated teleprotection equipment from ABB, Areva
( Al st om) and Si emens wi t h Al cat el -Lucent I P/ MPLS
communications equipment. Further tests have been conducted
with teleprotection equipment from ZIV/DIMAT, the TPU-1 with
E&M, X.21 and G.703 interfaces.
These tests again demonstrated that low end-to-end delay could
be achieved, that failover capabilities met or exceeded SONET/
SDH standards for tra"c re-routing (< 50 ms) and that total
control of the bandwidth required per application could be
achieved with an Alcatel-Lucent IP/MPLS network.
The capability of IP/MPLS networks to support teleprotection is
not only proven in the lab, it is proven in actual deployment.
Altalink, a Transmission Operator in Alberta, Canada with 11,800
km of lines and more than 300 substations, has successfully
deployed an Alcatel-Lucent IP/MPLS network in a live
environment since September 2010 supporting teleprotection
alongside general utility SCADA and other operational voice and
data tra"c.
More validation tests have happened in November 2012 with
Creos, the power utility of Luxembourg. The tests were meant to
validate the use of di!erential protection relays over the Creos IP/
MPLS network. The tests have proven that di!erential protection
works well over IP/MPLS, using C37.94 and E1 interfaces. The
tests were conducted over di!erent network paths; once over a
short path between two adjacent nodes and a second time over a
long path which consisted of 11 routers and a total ber length of
104km. The end-to-end latency observed during the tests was
3.87ms on the short path between the relays using C37.94
interfaces and 4.56 ms over the long path. With the E1 interfaces
on the di!erential relays, the end-to-end latency was 3.37ms over
the short path and 4.12ms over the long path.
The tests showed consistent results over a longer period of time
and prove that IP/MPLS is suitable for di!erential protection even
33
1 of 12
The ZIV-DIMAT TPU-1 was tested with E&M, X.21 and G.703 (E1)
interfaces.
Illustration 2.15 Teleprotection equipment successfully
tested over Alcatel-Lucent IP/MPLS networks
if the data is sent over a large number of hops. Each intermediate
hop between the ingress and egress hops adds 77microseconds
to the end-to-end latency.
Current di!erential protection schemes have been in operation on
the IP/MPLS network at Creos since January 2013.
34
Illustration 2.16 Di!erential protection testing at Creos
35
These tests all included at least 4 IP/MPLS routers in the path between the Teleprotection equipment, with congestion on the links.
(*): Tested on live power utility network between adjacent nodes.
Table 2.5.1 Summary of teleprotection over IP/MPLS performance tests
Vendor Model Type Latency (ms)
Jitter
(microseconds)
Interface
Siemens 7XV Distance 3.12 5 X.21
ABB NSD570 Distance 3.6 5 Ethernet
Areva DIP5000 Distance 3 5 X.21
ZIV/Dimat TPU1 Distance 5.92 E1
ZIV/Dimat TPU1 Distance 8 E&M
Nokia TPS 64 Distance 8.3 G.703 codir
Siemens 7SD52 Differential 3.12 5 X.21
Siemens 7SD52 Differential 3.87(*) C37.94
Siemens 7SD52 Differential 3.37 (*) E1
Areva P541/P591 Differential 3.25 5 X.21
Alstom P545 Differential 3.48 C37.94
Toshiba GRL100 Differential 2.9 3 X.21
Toshiba GRL100 Differential 0.1 3 Ethernet
36
In this particular case multiple T1 links are used from the microwave
radio to create an aggregate bundle of 12Mb/s. The protection
equipment is GE L90 using RS232 interfaces at 38.4 kb/s.
Illustration 2.17 Current di!erential scheme on 138kV
over a microwave link - tap to play
The example above also uses microwave links, albeit longer reach. The
payload is kept low which increases the packet rate.
Illustration 2.18 Current di!erential scheme on 500kV line
over microwave links
Examples of implementations
37
The secondary set of protection relays are SEL 311L and are connected
over G.703 interfaces at 64kb/s. The jitter bu!er setting is a bit more
narrow, hence the lower latency. The same microwave MLPPP bundle is
used as network link.
Illustration 2.19 Same link as the previous example with a
separate set of di!erential relays
In this example, there are two ber optic links between the nodes, one is
over OC-3 (155Mb/s) and the other is over a 10Gb connection. A
second distance protection scheme is used over a diverse path to
protect the same HV line. The second set of protection relays is Siemens
7SA522, also connected using G.703 interfaces.
Illustration 2.20 Distance protection scheme of 240kV line
over ber optic links
38
More complex and higher performance current di!erential protection schemes are
built in ring topologies as shown here.
Illustration 2.21 Four leg current di!erential scheme
Table 2.5.2 Master ring with E1 interfaces between A and B
E1
Interface
Jitter
Buffer
ms
Payload
Bytes
SITE B SITE A SITE A Site B
B to A A to B A to B B to A
CESoPSN 5 480 7.33 7.33 7.46 7.46
SAToP 1 64 1.15 1.15 1.26 1.26
Table 2.5.3 Slave ring with E1 interfaces between A and B
E1
Interface
Jitter
Buffer
ms
Payload
Bytes
SITE B SITE A SITE A Site B
B to A A to B A to B B to A
CESoPSN 5 480 7.46 7.46 7.53 7.53
SAToP 1 64 1.17 1.17 1.29 1.29
39
Table 2.5.4 Master ring with E1 interfaces. Readout of the 7SD52 relay latency in ms
E1
interface
Jitter
Buffer
ms
Payload
Bytes
SITE B SITE D SITE D SITE C SITE C SITE A SITE A SITE B
B to D D to B D to C C to D C to A A to C A to B B to A
CESoPSN 5 480 5.07 5.07 4.81 4.81 5.31 5.31 5.08 5.08
SAToP 1 64 1.21 1.21 1.11 1.11 1.10 1.10 1.16 1.16
Table 2.5.5 Slave ring with C37.94 interfaces. Readout of the 7SD52 relay latency in ms
C37.94
Jitter
Buffer
ms
Payload
Bytes
SITE B SITE D SITE D SITE C SITE C SITE A SITE A SITE B
B to D D to B D to C C to D C to A A to C A to B B to A
CESoPSN 5 32 4.00 4.00 3.87 3.87 3.93 3.93 3.93 3.93
SAToP 2 32 2.50 2.50 2.37 2.37 2.43 2.43 2.43 2.43
40
Extensive testing has been conducted at the Strathclyde university by Dr. Steven
Blair and Dr. Campbell Booth. The full test report is available on the Strathclyde
university website: Real Time Teleprotection testing using IP/MPLS over xDSL.
Illustration 2.22 Di!erential protection over xDSL with IP/MPLS
This chart illustrates the available bandwidth versus the distance on
typical copper lines depending on the DSL technology used.
Illustration 2.23 Bandwidth versus distance for xDSL
Section 6
Test your knowledge on communication networks for
Teleprotection
41
Review 2.1 Questions on Chapter 2
Check Answer
Question 1 of 8
Place the labels on the image where they t in terms of interface type
and typical speed
E&M
E&M
X.21
X.21
C37.94
C37.94
Chapter 3
Migration
Options
In the previous chapter, we have established that
IP/MPLS is ideally suited to replace the SONET/
SDH based TDM networks in use by power utilities.
The challenge is how to seamlessly migrate from
SONET/SDH to an IP/MPLS network. In this
chapter we will cover a number of possible
scenarios.
Migration options will vary depending on a number of factors, most importantly,
the availability of bandwidth for the IP/MPLS routers on the communications
infrastructure. In cases where unused ber optic cables are available in the cable
bundle that connects the substations, a parallel network can be created which
allows the most smooth migration scenario to be used. The picture below
illustrates how this could work.
Section 1
IN THIS SECTION
1. When spare optical ber is available
2. Reuse of SDH/SONET backbone
3. Introducing CWDM optical multiplexing
Migration options
43
Moving from SDH to an IP/MPLS network involves careful planning and depending on the
availability of optical ber, di!erent scenarios may be applied.
Illustration 3.1 Using separate ber optic cables in the same bundle
In cases where there is not extra optical ber available to create a
separate IP/MPLS network, the IP/MPLS network can be built as
an overlay on top of the existing SDH/SONET infrastructure. For
instance, if there is already an Ethernet over SDH/SONET
available at the substation, the IP/MPLS router can be connected
to that interface and immediately replace the router connected to
it at the substation. If there is no Ethernet interface available on
the SDH/SONET multiplexer, then the IP/MPLS router can be
connected through an STM-1 interface for instance. This would
enable IP and Ethernet services to be deployed at the substation.
Following that, other services like SCADA and Teleprotection can
be migrated across to the IP/MPLS platform.
Once this is completed, the TDM/PDH layer can be removed,
leaving the SDH/SONET multiplexer as the only legacy network
element in the network. In order to remove this SDH/SONET
element from the network gracefully, it is best to introduce a
WDM layer in the optical network. In many cases there is an
existing WDM infrastructure. Alcatel-Lucent o!ers a full range of
DWDM and CWDM solutions. The 1830PSS family of products
o!ers very advanced DWDM capabilities and is also fully
integrated within the Alcatel-Lucent 5620 Service Aware
Management sol uti on. Furthermore, the 7705 Servi ce
Aggregation Router product family o!ers an integrated (passive)
44
Illustration 3.2 Scenario 2, re-use of SONET/SDH
Illustration 3.3 Introducing WDM as a migration enabler
CWDM solution which can further simplify the migration process.
The illustration below shows how a 7705 SAR can be used initially
as a passive CWDM solution to initially provide more optical cable
capacity before the IP/MPLS capability is added.
There are several CWDM cards available for the 7705 SAR
products. The illustrations on the right show a couple of these
cards and how they can be used. The 4 channel card also
supports the 1310nm wavelength which is typically used in SDH/
SONET networks. This makes the 7705 SAR ideally suited to
integrate with existing SONET/SDH networks.
45
Illustration 3.4 Implementing CWDM with the 7705 SAR
Single color Passive CWDM Optical Add/Drop Multiplexer card (OADM)
for 7705 SAR-8 and 7705 SAR-18.
Illustration 3.6 7705 SAR CWDM modules
Illustration 3.5 Sample CWDM setup with Alcatel-Lucent
7705 SAR
7705 SAR-8/18
7705 SAR-8
7705 SAR-M
7705 SAR-8
7705 SAR-8 7705 SAR-8 7705 SAR-8
Chapter 4
MPLS
Technologies
for Teleprotection
In this chapter we analyze the di!erent options for next
generation packet networks to support mission critical
applications such as Teleprotection.
This chapter is meant to help power utilities to understand which
technology is most appropriate for their needs and to help
understand the basic di!erences between the proven technology
of IP/MPLS and more recent MPLS-TP.
IP/MPLS, contrary to what many people believe, is not a new
technology. Its development started in the mid 1990s in order to
improve the performance of routers. The idea of using labels
instead of IP addresses was driven by the limited performance of
routers to perform address lookups and their abilities to scale in
large networks. Since those days, the paradigm has shifted,
router performance no longer is the prime issue, however
network and service scaling is.
The telecommunications industry has been focusing on IP/MPLS
since the year 2000 and has created many standards ensuring a
fully featured and interoperable framework. IP/MPLS has become
the standard communications solution for service providers
worldwide and has found great success in mission-critical
networks in such industries as rail, electrical utilities, government
and public safety organization, defense and some large
enterprises.
Some communications vendors are in the early stage of MPLS-
TP consideration and implementation as those standards
become more formalized. Before any signicant uptake occurs
though, two crucial questions need answering:
# Is it a viable technology?
# Is it a compelling technology?
These are even more signicant for non-carrier, mission-critical
networks for organizations such as power utilities. Initially, the
principle driver for MPLS-TP was to apply a simplied subset of
IP/MPLS to bridge the gap between the packet and transport
worlds by combining the packet e"ciency, multi-service
capabilities and carrier-grade features of IP/MPLS with the
transport reliability and OAM tools traditionally found in SONET/
SDH. This seems like a reasonable approach on the surface but
as you will see, there are a number of drawbacks and limitations
associated with this simplied subset. In fact many of the
objectives of MPLS-TP have meanwhile been achieved by IP/
MPLS technology implementations.
Section 1
Market evolution of IP/MPLS and MPLS-TP
47
Alcatel-Lucent has already achieved the objectives of MPLS-TP
through its extensive IP/MPLS product portfolio and
management by the extensively proven 5620 Service Aware
Manager (SAM) network management tool. This end-to-end
management tool supports not only the Alcatel-Lucent IP/MPLS
portfolio of routers and switches but also the DWDM long-haul
optical transport products (1830 PSS) and packet microwave
radio products (9500 MPR) with extensive capabilities and time
saving, error preventing solutions.
The next sections in this chapter will describe the fundamental
distinctions between IP/MPLS and MPLS-TP, as it pertains to
mission-critical industries such as power utilities.
48
The concept of a unied communications framework based
around IP/MPLS consists primarily of two main functions. These
functions are
# Transport Layer Functions
# Services Layer Functions
There is a wide variety of transport layer functions used today in
the MPLS context, including LDP, RSVP-TE, BGP labels and
MPLS-TP.
For the purpose of transport, it is really a matter of comparing the
applicability of RSVP-TE and LDP to static MPLS-TP as a
transport mechanism linking to the service layer functions. This
service layer function, driven by applications, is converging on IP.
This is true in the railway metro and mainline market segments,
for example, even though some TDM based applications remain
(GSM-R, Interlocking, SCADA, operational telephony...). The
same applies to utilities, where the applications are rapidly
converging on IP (IEC104, Goose, eSCADA...) but where some
TDM applications will remain for a long time (Teleprotection,
SCADA...).
IP/MPLS
There are two major control plane protocols that allow the
creation of an IP/MPLS service network, which are RSVP-TE and
LDP. RSVP-TE provides many tools to achieve the same level of
control on the network as an SDH network, where LDP provides
a much more dynamic environment more equivalent to a full IP
network (with IP network we mean typically a connectionless IP
network). It is important to note that both technologies can easily
co-exist on a single network and that it is at the design stage that
one or the other protocol will be used to answer the needs of the
respective applications in terms of resiliency and control. This
exibility may depend upon the vendors implementation in their
respective products.
The development of RSVP-TE based tra"c engineering has been
ongoing for more than 15 years and is still evolving today to meet
the needs of ever converging applications. This technology is
widely deployed and very successful in meeting high availability
demands. This includes the functionality to facilitate Dynamic
Control Planes for automatic bandwidth adjustments and re-
routing or optimization, as well as protection and rerouting
Section 2
MPLS Technologies
49
mechanisms. The control plane used for facilitating these
functions is indirectly dependent on the presence of an IP based
control plane. The IGP will update the tra"c engineering
database (TED), with all the CSPF (Constrained Shortest Path
First), link coloring information etc and RSVP-TE will then utilize
the information in the TED. This architecture creates an
abstraction between the IGP and IP/MPLS meaning that a failure
in the IGP will not cause RSVP-TE based paths to fail.
It is within this market that Alcatel-Lucent has pioneered the
adoption of the use of this IP based control plane to make use of
RSVP-TE and LDP based transport mechanisms to facilitate the
establishment of a framework for delivering both Point to Point
and Point to Multipoint L2 Services. In addition to the use of
these mechanisms, Alcatel-Lucent has also been able to enhance
the L2 point to point and multipoint services with L3, L4 and
application level intelligence. This allows critical networks to
continue to evolve the services on the installed platforms as the
industry demands.
This exibility that ALU has brought to the market is
unprecedented and fundamentally based on the in-house based
data path hardware and software developments allowing the
continuous adoption of new technologies and standards as the
industry evolves. Only this approach can ensure investment
protection for periods exceeding 10 years.
IP/MPLS For Mission-Critical Networks
A key enabler for the safe and e"cient transformation of power
utility communications network is a modern, reliable and exible
infrastructure that forms the core network in order to route the
critical application tra"c such as Teleprotection, grid monitoring,
control and status and key corporate data e!ectively, e"ciently
and on time. Furthermore, non-critical applications around
substation automation services such as voice, video surveillance,
corporate LAN/ or VPN and even public internet, are also able to
leverage the same infrastructure. These types of requirements
are forcing many industries, governments and enterprises to
50
The Multi Protocol nature of IP/MPLS means that it can run on any
transport network, over any media and it natively supports any
protocol on top of it.
Illustration 4.1 IP/MPLS supports all protocols and
any transport mechanism
consider an evolution of their communications infrastructures that
would be very di!erent from their traditional Time Division
Multiplexing (TDM) centric networks. A exible transformation is
required to preserve existing investments and to minimize risks.
Al catel -Lucent IP/MPLS communi cati ons i nfrastructure
incorporates state-of-the-art technologies to enable an industry,
government or enterprise to deploy a future-proof, highly
available IP network to continue supporting existing TDM and
legacy applications while providing a smooth migration path to IP,
Ethernet and IP/MPLS-based services. This new IP/MPLS
infrastructure will maximize the cost-e!ectiveness and e"ciency
of the network without jeopardizing reliability, while enabling the
deployment of new devices and applications that can improve
operational and workow e"ciency.
A highly available IP/MPLS communications infrastructure is
ideally suited to support both mission-critical operations and
corporate communications requirements. In addition, the Alcatel-
Lucent network and service aware management platform allows
organizations to improve their e"ciency by automating and
simplifying operations management for communications services,
thus reducing the barrier in introducing IP/MPLS-based
technologies and services.
MPLS-TP
The I nter net Engi neeri ng Task Force ( I ETF) and the
Telecommunication Standardization Sector of the International
Telecommunication Union (ITU-T) have undertaken a joint e!ort to
standardize a new transport prole (TP) for the multi-protocol
label switching (MPLS) technology that is intended to provide the
basis or the next generation packet transport network. The
fundamental idea of this activity is to extend MPLS where
necessary with Operations, Administration and Maintenance
(OAM) tools that are widely applied in existing transport network
technologies such as SONET/SDH or OTN and to enable it to
operate in the absence of a dynamic IP-based control plane.
MPLS-TP, as dened in RFC5921 in the IETF, is positioned to
enable MPLS to be deployed in a transport network and operated
in a similar manner to that of existing transport technologies. It
enables MPLS to support packet transport services with a similar
degree of predictability, reliability and OAM to that found in
existing transport networks. It denes a subset of MPLS
protocols for Layer 1 (L1) and Layer 2 (L2) only.
Some of the rationale behind the push for MPLS-TP is that there
is an increasing demand for legacy transport networks to support
packet based services and hence the need for evolution to MPLS
like behavior, as an evolution of SDH. IP/MPLS networks are
perceived as complex by some in the transport world and there is
sometimes a further requirement to keep the layer 2 and layer 3
networks separate. In many cases the existing SDH infrastructure
and transport switches lack native IP support. The alternative to
providing native IP support is to combine the architectural,
51
management and operational models of Circuit Switched
transport networks with Packet switching optimizations using an
MPLS data plane and additional OAM and Protection capabilities.
The main arguments that are generally used to promote MPLS-TP
are the following:
# Transport solution leveraging SDH experience and therefore
with no control plane, leading to a fully manual provisioning
# High performance set of OAM and protection tools to allow
fast failover times
# Capacity to integrate legacy tra"c inherited from SDH
evolution.
# Bi-directional transport of data in LSPs reducing the risk of
path mismatch between a source and a destination and its
return path.
# Reduced cost of equipment (CAPEX), as the technology
requires less intelligence due to the absence of a control
plane.
# High level of security due to the absence of IP based control
plane.
These arguments need to be put in perspective when comparing
to IP/MPLS technology and implementations, but also when
determining the applicability of MPLS-TP for industry
applications. In short, these are the counter-arguments for IP/
MPLS.
# IP/MPLS can deliver very simple transport solutions just as well,
however they o!er the advantage of allowing very rapid and
exible changes (adding nodes, links, applications) which is a
lot more cumbersome with SDH and MPLS-TP.
# Alcatel-Lucent o!ers even more OAM tools as part of its IP/
MPLS solution not only to assure fast failover times in the
transport layer, but also to assure the services at higher layers.
# The capacity to integrate legacy tra"c is no di!erent in MPLS-
TP from IP/MPLS, both have CESoPSN and SAToP capabilities.
# The bi-directional LSP behavior is guaranteed by Alcatel-
Lucents 5620SAM management tool, so there is no divergence
of forward and reverse LSP paths possible.
# The idea that MPLS-TP reduces CAPAEX is a major mistake
because it is forcing another layer 3 overlay to be added to the
network which is not the case with IP/MPLS.
# The control plane protocols used in IP/MPLS are not part of the
MPLS labeled tra"c for the applications, they are secured and
authenti cated. Furthermore control pl ane protecti on
mechanisms are in place to prevent any control plane attack to
have any impact on the user/application tra"c.
52
Building a mixed network with IP/MPLS and MPLS-TP
technologies
The relationship between IP/MPLS and MPLS-TP can be
depicted in this diagram (used at the MPLS World Congress)
where MPLS-TP was promoted for large access/aggregation
networks for transport scalability while IP/MPLS remains in edge
and core for services, however interest for MPLS-TP at MPLS
World Congress has subsided.
The rst information that can be extracted from this drawing is
that carriers consider MPLS-TP as a backhaul technology, but
still consider that all the services part is to be provided by IP/
MPLS. Whether this vision applies to private networks which are
typically smaller than large carrier networks is very questionable.
Are private industry organizations willing to train two teams on
two di!erent technologies, quite possibly from two di!erent
vendors if not necessary? Such a choice has a very high OPEX
cost and probably does not apply in markets other than mobile
operators.
From a technical perspective, there are also several challenges
when interworking IP/MPLS and MPLS-TP domains. The rst is
that there is no control plane interworking between the two.
53
MPLS-TP is being positioned as the aggregation network for large
carrier networks.
Illustration 4.3 End to end vision of large carrier networks
including MPLS-TP
Providing end-to-interworking between MPLS-TP and IP/MPLS is quite
challenging because of the di!erent control plane architectures, disjoint
protection mechanisms and di!erent Operations Administration &
Maintenance (OAM) tools.
Illustration 4.2 End-to-end interworking challenge
The second is that end-to-end interworking is very di"cult in a
multi-vendor environment and no mechanisms to ensure end-to-
end resilience interworking as shown in Illustration 4.3.
Note also that from an operations perspective, there is little
chance to nd an end-to-end network management tool because
MPLS-TP vendors often do not have IP/MPLS solutions and IP/
MPLS vendors which may have MPLS-TP solutions generally
have two di!erent management solutions.
Alcatel-Lucent however have both IP/MPLS and MPLS-TP
solutions, managed by the same management system, 5620
Service Aware Manager.
54
The 5620 SAM manages both IP/MPLS and MPLS-TP.
Illustration 4.4 Screen shot of 5620 SAM
This section will review the positioning used to promote MPLS-
TP into certain markets, demonstrate how IP/MPLS can meet at
least the same level of functionality of MPLS-TP and review other
values of IP/MPLS that cannot be met by MPLS-TP.
This section will also describe how Alcatel-Lucent has enhanced
the IP/MPLS standard to provide a unique, secure, complete
end-to-end solution.
Comparison between MPLS-TP and IP/MPLS technology
# Manual provisioning: Some applications like teleprotection in
utilities and SCADA in many di!erent industries, built on the
blue/red model, require the avoidance of common mode failure
scenarios. For that matter, in the case where a single physical
network is built (typically a ring), there is a need to control the
tra"c between the source (SCADA RTU) and destination
(SCADA MASTER), in order to ensure that they dont share a
common path or equipment. MPLS-TP being built around
manual provisioning can provide this feature, but, IP/MPLS with
the use of RSVP-TE (tra"c engineering), can also provide
exactly the same feature. However, the provisioning process is
easier with IP/MPLS due to the fact the provisioning can be
static end-to-end as in MPLS-TP but can also be a mix of strict
and loose therefore alleviating the load of provisioning required.
This solution has been validated by several teleprotection and
SCADA vendors.
# High performance set of OAM: IP/MPLS has many OAM
features that are similar to that of MPLS-TP. The only di!erence
is in the way it is used, but overall, it provides the same
services. The OAM trigger resiliency features available as part
of IP/MPLS (Fast ReRoute, primary/secondary LSP, active/
standby pseudowire) can also be used to control SLAs. Table
4.4.1 below lists all OAM tools which are available on the
Alcatel-Lucent IP/MPLS platforms. Bi-Directional Forwarding
Detection (BFD) can be used to trigger fast failover of L3
protocols such as RSVP, VRRP at 10ms intervals.
# Capacity to integrate legacy tra"c: The legacy transport
(typically E1) is part of the MPLS standard (CESoPSN and
SAToP) and is therefore supported by IP/MPLS and MPLS-TP.
Main mobile operators are successfully transporting E1 tra"c
of GSM over IP/MPLS for more than 5 years, demonstrating the
Section 3
Comparing technologies
55
technology is totally suited for such legacy transport.
# Bi-directional transport of data in LSPs: In order to totally
eliminate the risk, for applications that are sensitive, IP/MPLS
RSVP-TE supports static provisioning of paths which allows
control of tra"c end-to-end and ensures that bi-directional
tra"c takes the same path, exactly as MPLS-TP does.
Furthermore Alcatel-Lucent 5620SAM will always provision
symmetrical LSP paths.
# High level of security: IP/MPLS has been deployed by carriers
for years and has been demonstrated as a very solid technology
to protect data. However, as with all security frameworks, it is
not only a matter of technology, but also a matter of process
and implementation, to ensure full security. The utility market
have rolled out a specic security framework called NERC-CIP
(in North America) to protect their mission-critical applications.
It is important to note that many utilities networks world-wide
have successfully met these criteria with an IP/MPLS network.
Many critical Defense networks are using an IP/MPLS
infrastructure and have so for years.
# Platform cost: Is an IP/MPLS platform more expensive than an
MPLS-TP platform? Both can use the same kind of silicon
technology, so, there is no reason for a price di!erence.
Therefore, this question needs to be looked at in the broader
context of network requirements on the platform itself, not only
in the context of technology. For example:
56
Table 4.3.1 OAM tools on Alcatel-Lucent IP/MPLS routers
OAM function
ICMP & ICMPv6 Ping Traceroute
Two Way Active
Measurement
TWAMP
LSP diagnostics LSP Ping, LSP traceroute
SDP diagnostics SDP ping, SDP MTU path discovery
VLL diagnostics VCCV ping, VCCV trace
VPLS MAC diagnostics
MAC ping, MAC trace, MAC populate, MAC purge,
CPE ping
Ethernet OAM 802.1ag, Y.1731, 802.3ah
Ethernet loopbacks Line, internal
OAM propagation to
attachment circuits
ATM ports, E1 ports, Ethernet ports, Pseudowire
status signaling propagation
LDP signaling
Multicast debugging Multicast trace, Multicast stats, Multicast route info
LDP Tree trace Allows to see multiple paths for a given service
Service Assurance Agent SLA monitoring
Control Plane Assurance
Management
OSPF, ISIS, BGP, PIM,... network control plane
visualization
BFD Bi-directional Forwarding Detection
Is the capability to absorb tra"c burst to minimize packet
loss crucial to the operational applications in mission-critical
networks? If yes, then the memory cost would be the same on
both platforms.
Is supporting a large number of OAM sessions (e.g. Y.1731
continuity check) with a small transmission period crucial to
achieve SDH/SONET-like fault detection speed? If yes, then the
platform requires either a powerful processor or specic hardware
assist to process in-service and out-of-service OAM protocol
data units (PDU) continually for tasks such as frame loss
measurement and delay/jitter measurement in a scalable fashion.
MPLS-TP Continuity Check does require this.
Is accurate synchronization through packet-based
synchronization transport mechanism like IEEE1588v2 important?
If yes, the platform must make use of specic hardware to be
able to insert and extract timestamp information with enough
precision. Also a high quality oscillator needs to be used to
minimize frequency optimize synchronization recovery.
With the advent of next generation silicon and processor
technology, packet processing and forwarding (switching,
bridging or routing) in the data plane is no longer a dominant cost
factor when designing a new platform. When comparing
platforms of di!erent packet-based technologies, close attention
must be paid to understand whether the platforms meet other
non-data plane requirements, such as those listed above, which
are equally pivotal when building a mission-critical network.
When all factors are considered, the platform cost di!erence
should be negligible.
As well, since IP/MPLS supports layers 1 through 3 and MPLS-TP
only supports layers 1 and 2, there is signicant cost to having
separate layer 1 and 2 from layer 3 solutions including equipment,
management, training, sparing, etc. This is discussed further in
the document.
Other values brought by IP/MPLS
As mentioned, MPLS-TP generally ignores the end to end
requirements, mainly for IP applications and this is where IP/
57
Because of its native IP capability, IP/MPLS can o!er a more
comprehensive cybersecurity solution to mission critical networks.
Illustration 4.5 Cybersecurity enabled by IP/MPLS
MPLS fullls the complete range of services for mission-critical
applications.
# IP Awareness:
Typically MPLS-TP has no IP awareness and therefore, for all
applications that are IP based (eSCADA, CBTC, VoIP, meter
reading, CCTV, PMUs...) customers using an MPLS-TP path will
have to add an IP layer on top. Therefore, the network will look
like the one in Illustration 4.7.
Such architecture creates issues because the two layers operate
independently meaning that there is absolutely no dynamic
collaboration between the two layers. Main concerns are the
following:
# MPLS-TP provides poor e"ciency of bandwidth because there
is a need for native IP tra"c to be transported between the two
layers and typically the end to end path will likely not be
optimized because it is xed.
58
Whatever the application requirements at L2 or at L3, IP/MPLS supports
all of them, either for point-to-point or point-to-multipoint applications.
Illustration 4.6 IP/MPLS supports all services at L2 and 3
Using separate technologies for aggregation and edge plus core leads to
more complexity and is more di"cult to manage.
Illustration 4.7 Overlay IP network on top of MPLS-TP
Transport
# End to end resiliency for IP applications will be triggered by
MPLS-TP features and Layer 3 protocols, typically OSPF,
therefore, convergence times can be much higher than 1
second in many scenarios which may create big outages for IP
applications such as operational telephony over IP. This can
create impacts on latency in the case where the lower layer
backup scenarios are di!erent than the upper layer scenarios.
# Provisioning of services will be considerably more complex as
there is a need to fully mesh the IP routers which will typically
require a lot of LSP manual provisioning.
# Troubleshooting will be more complex as there is a need to use
di!erent tools and di!erent technologies.
# Most private network operators need to segment their IP
applications in di!erent networks for organizational or regulatory
issues including CCTV, intranet, telephony, nancial payment
applications, etc. With a basic IP network, this is not possible
unless using complex VRF features which, again are very
statically congured hop by hop and this activity doesnt scale
well.
# Most vendors of MPLS-TP have little IP knowledge and
therefore railway, utility, government and other organizations will
have to deal multiple vendors and their respective network
management tools which can lead to a lack of clear ownership
when a problem arises. IP/MPLS natively understands IP tra"c
through IP-VPNs and therefore can ensure through a single
technology (single vendor and, potentially, single management
tool) a fully optimized transport of data end to end. That is the
typical solution that has been used by carrier networks for more
than ten years and has recently become the solution of choice
for utilities, government organizations, railways and other
mission-critical networks.
# Fast provisioning of services: mainly applications such as
SCADA, Phasor Measurement Units, GOOSE... are not only
point to point but becoming more point to multipoint and often
require the exibility for disaster recovery scenarios. Therefore,
doing manual provisioning like MPLS-TP for such applications
doesnt scale at all. That is why for these applications, having
the choice to adapt provisioning from very manual to fully
dynamic, as IP/MPLS provides, answers all needs of all
applications. MPLS-TP can only provide solutions for some
challenges of some applications but typically does not take a
real end to end view of all requirements for railway, utilities and
defense applications in a network.
# Multiple failure scenarios: IP/MPLS supports many scenarios for
resiliency and always ensures that as long as there is a path
available between a source and a destination, the tra"c may
use it, if it makes sense for the application. MPLS-TP has only
one alternate path, if the rst one fails, which may not be
enough to reach 99.999% availability expected by mission-
59
critical and safety applications. It is important to note that in
very meshed networks, ensuring that the primary LSP does not
share any path or device with its redundant LSP is not always
that easy to control. Relying on manual provisioning or on
unproven network management tools can be very risky
compared with the much proven and dynamic technology of IP/
MPLS, with LDP for example, that has years of e!ective
demonstration of resiliency capabilities.
# Multicast Awareness: the number of video cameras is
increasing heavily in many organizations for CCTV and is the
most bandwidth hungry application that may be transported
over a mission-critical communications network. In order to
optimize any video tra"c delivery, IP has been enhanced with
multicast protocols which have a simple role of ensuring that a
video stream is transported once over a physical link
independently of the number of receivers being registered for
the stream. The multicast protocols basically use the di!erent
network equipment (routers, switches) to replicate the tra"c.
While IP/MPLS has native multicast capabilities, MPLS-TP has
none, which means that for example if a CCTV camera stream
has to be stored in two locations (for resiliency) and has to be
monitored by two OCCs, the tra"c of each camera will be sent
four times over the links in access and core. Multiply this by
3Mbps, which is the average bandwidth for good quality video,
you realize that with 100 cameras for example, you will generate
900 Mbps more tra"c with MPLS-TP than with IP/MPLS.
Clearly, a major impact on the cost of equipment and
bandwidth, a fact often not mentioned by promoters of MPLS-
TP technology.
Alcatel-Lucent added value to a standard IP/MPLS
implementation
To make a clear and fair comparison between MPLS-TP and IP-
MPLS, organizations should also include in their study all
enhancements that have been made to IP/MPLS standards by
some specic design or implementations which are typically a
demonstration of a more mature technology and which applies to
60
MPLS-TP is not able to cope with multiple failures which is problematic
for critical infrastructure networks at times of natural disasters.
Illustration 4.8 IP/MPLS can cope with multiple failures
railway, utility, public safety and other organization running
mission-critical networks.
# Legacy integration: where most of MPLS-TP vendors only
transport E1, Alcatel-Lucent has developed natively in its
access router portfolio, some interfaces to cope with many
applications which use E&M (operational telephony and LMR),
FXO/FXS (for internal telephony), serial interfaces X21, RS232,
V35 (for SCADA), G.703 (for interlocking), C37.94 (for
teleprotection). This allows organizations to install a single piece
of equipment capable of connecting all legacy devices as well
as all IP applications. Compare this with MPLS-TP solutions
that typically would need the MPLS-TP switch, plus an IP router
plus a PDH mux. In those cases, the low price argument of
MPLS-TP is no longer relevant and CAPEX and OPEX (3
training, 3 management solutions, 3 contracts, 3 spare lots....)
will be in favor of an IP/MPLS solution.
# Synchroni zati on: Al catel -Lucent has devel oped many
synchronization capabilities into the IP/MPLS portfolio and can
claim support equal to or better than SDH, which is increasingly
being replaced by IP/MPLS networks. Relying on years of
deployments in mobile networks and in GSM-R networks
ensures that all features and tools to control clocking are fully
embedded for all applications such as GSM-R, Tetra,
teleprotection, IEC61850, etc. MPLS-TP solutions cannot
support such a variety of synchronization options that include
native GPS, native BITS ports, SyncE, 1588v2 and ACR.
# End to end application resiliency: most of the IP devices are not
directly connected to an MPLS router, but rather to an Ethernet
switch. In order to ensure end to end resiliency in line with
critical applications requirements, Alcatel-Lucent has developed
features that ensure that the rst IP/MPLS node traversed (the
PE) can be made redundant and that resiliency is not triggered
by Spanning Tree which is not only hard to manage, but also
very slow to converge. Features such as MC-LAG (Multi-
Chassis Link AGgregation) allow a switch, or server to connect
to two di!erent IP/MPLS routers to provide fast resiliency. This
feature provides 250 ms failover time. For trackside switches in
railways for example, the architecture leverages the standard
ring protocol G.8032 and in the same way, ensures end to end
resiliency between this access technology and IP/MPLS. As per
MC-LAG, resiliency in that case is ensured in much less than
250 ms. We have observed convergence times of less than
50ms with G.8032. Other technologies are available to ensure
resiliency with non standard protocols and all these features,
allow network operators to dene a real end to end architecture
that answers the requirements of all applications, only available
through IP/MPLS
# Richness of transport technologies: many critical network
operators (railways, air tra"c control, power utility, oil & gas,
61
defense...) can leverage their own bre infrastructure but for
resiliency purposes they sometimes have to create loops and
dont have bre to close the ring. In such cases, the native
implementation of microwave IDU in the IP/MPLS router
portfolio provides a very good solution to provide a seamless
end to end solution, with microwave awareness. In other
situations, bre has become a scarce resource and in those
cases, optical multiplexing is becoming necessary to provide
the expected bandwidth. In these cases, Alcatel-Lucent
provides a full end to end portfolio of optical CWDM and
DWDM sol uti ons whi ch are control l ed by the same
management product as the IP/MPLS and microwave products.
As well, some of the IP/MPLS routers have integrated passive
CWDM multiplexers to provide an optimized and small footprint
solution.
# Easy Network Management: MPLS-TPs claim that IP/MPLS is a
complex technology requires customers to look at the
operations of a full end to end communication network before
deciding. The Alcatel-Lucent 5620 SAM solution provides
complete end to end management further simplifying operations
and maintenance of an IP/MPLS solution beyond that of MPLS-
TP and certainly beyond requiring separate solutions and
management for layers 1 / 2 and layer 3.
# Provisioning is easy and the time to service is very short
compared to MPLS-TPs tedious manual provisioning. For IP
based applications, the full integration provides a much quicker
time to solve potential problems compared to an end to end
network with di!erent products and technologies.
62
In the mission-critical market segments of rail, utilities, public
safety, etc., most of the new applications rely on IP, even for
operational telecoms: eSCADA, signaling, teleprotection, LMR
such as Tetra, 3G and LTE mobile communications and a lot of
applications that include video such as video surveillance. All of
these IP applications wont be carried e"ciently on a transport
centric infrastructure, creating an additional burden on
provisioning to compensate for the ine"ciency of the transport.
IP/MPLS however, handles both services and transport, reducing
the complexity of a multi-technology model. This is depicted in
the diagram below where it shows that IP/MPLS provides both
the service intelligence as well as the necessary OTN interface to
the Transport network.
Typically, MPLS-TP suits the needs of a very hierarchical, large
network and has therefore been adopted mainly by large mobile
operators in China. In this scenario all communications are built
Section 4
IP/MPLS or MPLS-TP for mission critical applications
63
IP/MPLS covers the transport as well as the services while MPLS-TP
only covers the transport part. Alcatel-Lucents interworking with the
optical layer makes it possible to manage the service, transport and
optical layers from a single management platform.
Illustration 4.9 End to end service with IP/MPLS and
MPLS-TP technologies
64
Table 4.4.1 A comparison of the applicability of IP/MPLS and MPLS-TP for network transformation
Attribute IP/MPLS MPLS-TP
Maturity
proven for more than 10 years in carriers and with many
railway deployments, (typically pushed by UIC group),
utilities and public sector networks for government,
Defense, Public Safety, ...
few deployments in large mobile service provider networks
(mainly Asia). Infancy, little or no proven deployment
experience for railways, utilities and government networks
Standardization fully standardized since 1997 some still in progress
Multi-vendor
interoperability
proven for more than 10 years in many customers with
different vendors in the core, edge and access
limited to trials
TDM transport yes, CESoPSN or SAToP yes, CESoPSN or SAToP
VPN support Layer 2 and Layer 3 Layer 2 only
Service support full services transport only
IP Multicast delivery fully supported and optimized none
Protection Fast ReRoute, active/standby
Resilient to multiple
failures
YES NO, would require a control plane protocol
Trafc Engineering dynamic, easy to accommodate change static, difcult to cope with changes
Convergence of
transport and services
network
single solution covers all layers, less hardware diversity,
less pares, common management, one training
more hardware diversity, separate management systems, more
training to attend
OAM single platform from L1 up to L3 (and above) different platform needed for L3 , MPLS-TP covers L2 only
IP/MPLS & MPLS-TP
interworking
unproven unproven
manually and therefore would not support large scale any to any
service connectivity. IP/MPLS would be required for scalable
layer 3 any to any connectivity. Therefore, the organization would
need to maintain two technologies, which can only be justied by
very large organizations.
Some MPLS-TP users have already moved away from this
technology, towards a full IP/MPLS implementation because they
have come to realize that multiple network failures can occur and
at times of crisis such as natural disasters, the resilience of the
network is of utmost importance.
65
Conclusion on IP/MPLS vs MPLS-TP
Power Utility network operators who are considering to build new
communication infrastructures for their applications should
compare technologies in the context of end to end application
needs not in terms of a subset of the solution considering just
the transport infrastructure.
The most important thing to consider is that IP/MPLS does
provide the same services as MPLS-TP, but also much broader
services such as IP optimized transport, much easier
provisioning, larger exibility to accommodate di!erent
application requirements and more.
From an industry point of view, in the railway industry for
instance, the UIC has clearly pointed to IP/MPLS as a technology
to accommodate the needs of railway applications, MPLS-TP is
not considered a viable solution to fulll the end to end needs of
railway operations. In the last 5 years, many utilities have also
transformed their network to prepare for Smart Grid applications
which are typically IP based and more and more Any to Any
types of communications. Therefore, it has become clear to all
utilities that IP/MPLS was the way forward and that MPLS-TP
could not provide the adequate infrastructure for all applications.
Other users of MPLS-TP have realized that their infrastructure
should be resilient to multiple failure scenarios which often occur
when natural disasters happen such as oods, earth quakes etc.
These customers have chosen to move to IP/MPLS for this
important reason.
In fact a lot of these utilities are taking the IP/MPLS philosophy
from the WAN deeper into the network, into the substation LAN.
In this way, IP/MPLS ensure seamless provisioning of all services
down to the application inside the substation.
All in all, comparing the installed base, technology maturity,
development investment, standards evolution, market knowledge
and adoption, critical network operators should nd in IP/MPLS
not only the technical solution but also the market to allow them
to securel y depl oy mai ntai n and grow thei r network
infrastructures for decades to come.
Section 5
Conclusion on MPLS
66
Chapter 5
The Importance
of
Synchronization
In previous chapters we have addressed the need
to support legacy interfaces and the requirement of
accurate synchronization throughout the network. In
this chapter we will cover the synchronization
options provided by packet networks, the accuracy
they deliver and their application in power utility
networks.
This chapter is mainly based on the Synchronization over
Ethernet Networks white paper from Alcatel-Lucent.
Many applications in power utility networks require an accurate
synchronization mechanism. This can be required in order to
provide timestamp information to make sure that the application
itself can assess the validity of the data. Especially when
samples are made of current and voltage in the power grid, it is
important to know exactly at what time this data was recorded in
order to make the right decision about the status of the power
grid at that point in the network. SONET and SDH technologies
have by the very nature of the technology had built-in
synchronization capabilities, the S in the acronym stands for
Synchronous. In every SONET/SDH network where an accurate
time-stamping capability is required, there is one master clock
source (or more than one for redundancy) which is used to
propagate timing information to all nodes in the network.
In the past, packet networks have lacked the capability to
provide timing accuracy which is required in real time network
environments of power utilities, railway operators and mobile
service providers. They lacked the accuracy, resiliency and
scaling required for these networks.
Over recent years, several (packet based) technologies have
emerged to meet these timing needs. These range from changing
the Ethernet physical layer to provide synchronization reference
distribution (along the lines of SDH and SONET) to Timing Over
Packet (ToP) techniques such as Adaptive Clock Recovery (ACR)
on Circuit Emulation Services, enhanced NTP and IEEE1588v2.
The latter two technologies also address the requirement for the
distribution of highly accurate time for applications that need
time-of-day or phase accuracy.
When power utilities, railway operators and service providers
look toward these new technologies, they need to view the
transport of timing and time as a network service. These
services will have network planning considerations along with
OAM&P tools in order to ensure proper operation within the
network.
Traditionally, reference timing has been distributed using
synchronous physical layer technologies. This means that the
Section 1
Introduction
68
devices driving the interfaces have the ability to both transmit the
data using a timing reference and to recover the timing from the
received data for reference purposes. Many technologies include
this capability. For example: PDH, SDH, SONET, POS, PDH
Microwave, DSL, GPON and WDM. Traditional Ethernet does not
fall into this class since it uses local interface timing rather than a
timing reference for its transmissions. The Ethernet interfaces
either need to be modied to act as synchronous physical layer
interfaces (Synchronous Ethernet) or a higher layer timing
distribution protocol needs to be used (Timing Over Packet).
These two approaches are depicted below.
Synchronous Ethernet, which was introduced in 2008 is an
enhancement of a legacy Ethernet interface and includes the
ability to relay accurate timing information along with the Ethernet
frames over the physical media. It is the highest performing
solution for timing over Ethernet, but may cause some issues for
complete network rollout. For instance when a part of the network
does not have the hardware capability to support Synchronous
Ethernet, because it is going over another service provider
network for instance.
Several ToP techniques exist that allow for the transport of timing
information without the need for a synchronous physical layer.
These include NTP, IEEE1588 and ACR. Network Time Protocol
(NTP), dened in IETF RFC 1305, has been used for many years
to allow distributed devices to synchronize with respect to Time
of Day. It uses timestamps embedded in packets to accomplish
this synchronization. This protocol has been deployed in
networks for over 20 years and has been proven to provide
reliable time distribution to accuracies in the order of
milliseconds. This level of accuracy is often acceptable for alarm
time-stamping, billing and statistics; however, for applications like
E1/T1 timing and mobile base-station phase and frequency
references, accuracies of approximately 1 microsecond are
69
Synchronous Ethernet and Timing over Packet are two di!erent
mechanisms that can be used to provide synchronization in Ethernet
networks.
Illustration 5.1 Two main principles of synchronization
required. IEEE1588 has, at its core, a timestamp distribution
mechanism very similar to NTP. Version 1 was targeted at time
distribution over an Ethernet LAN while Version 2 (introduced in
2009) adds some key capabilities to address the distribution
over the WAN environment. Both versions were designed to
allow higher accuracy time distribution than possible with
NTP.
Circuit Emulation Service (CES) implementations use the
constant bit rate of the TDM interfaces to generate packets.
Adaptive Clock Recovery relies on this constant rate of
packet generation to recover the original timing information.
This is primarily intended to allow for the transport of the
originating E1 or T1 service clock, but if the originating
interface is known to be locked to the PRC traceable
reference, CES can then be used to distribute a highly
accurate timing reference.
In the following sections, we will discuss these di!erent
options that are available to ensure proper synchronization of
the network and provide support for Teleprotection, SCADA
and PMU applications on a converged IP/MPLS network.
It is important to understand the di!erent techniques
available in order to achieve network synchronization.
Implementing the right synchronization solution will make the
transition from the legacy installed base to a new network a
lot easier.
70
During early 2006, several European Telecom companies started
an initiative within the ITU-T to dene the requirements for having
the traditional Ethernet interfaces meet comparable timing
performance targets to those of SDH/SONET interfaces. They
recognized that the Layer 1 relaying of synchronization
information would potentially be the most reliable form of
synchronization transfer and would not have any impact from the
packet delivery load over the interface.
By deploying a network of Synchronous Ethernet interfaces, or a
hybrid of SDH/SONET and Synchronous Ethernet, the network
provider can ensure the delivery of the same quality of timing
references through the network as is currently achieved using
SDH/SONET only. This can then be used for TDM services at the
network edge or as a timing reference into Mobile base-stations
for carrier frequency derivation.
Synchronous Ethernet uses the physical layer of the Ethernet link
to distribute the clock among nodes in the network. In an
analogous manner to SONET/SDH, each node has a local or
system clock which determines the outgoing clock rate of each
interface. The system clock is derived from the incoming
clock at one of its input interfaces or from a dedicated timing
interface such as a BITS port. Synchronous Ethernet is based on
the same architectural structure as SONET/SDH. It is important
to note that Synchronous Ethernet works at layer 1 and is
concerned only with the precision of the timing of signal
transitions to relay and recover accurate frequencies. It is not
impacted by the tra"c load. For this reason, it has been shown
to have a performance equivalent to that seen in SDH/SONET
networks.
Illustration 5.2 shows a test environment created within the
Alcatel-Lucent laboratories to analyze the performance of a long
chain of a total of 20 Service Routers and Service Aggregation
Routers using Synchronous Ethernet as the reference frequency
distribution method. For this test, six of the devices were
deployed within a thermal chamber to introduce extreme
temperature variations into the environment. Even with this long
chain and a 65 C temperature range, the timing reproduced at
the end of the network was two orders of magnitude better than
the network limit as dened by the ITU-T as is illustrated in
illustration 5.3.
Section 2
Synchronous Ethernet
71
The Synchronous Ethernet specication also includes the same
ability to relay timing source quality level information as is found
in the Synchronization Status Messages (SSMs) of PDH and
SDH/SONET. Since Synchronous Ethernet uses Ethernet OAM
messages for this purpose, there is discussion within the
standards bodies of these SSMs being expanded to support
many new features to assi st i n the management of
Synchronization Distribution. Alcatel-Lucent is actively involved in
these discussions and is developing applications to make use of
these capabilities.
72
Synchronization quality tests were conducted over 20 nodes, 6 of which
were placed in a thermal chamber with temperature changes of 65C
over more than 24 hours
Illustration 5.2 Synchronous Ethernet test over 20 nodes
The chart shows how well the Synchronous Ethernet technology
exceeds the ITU G.823 and ANSI DS1 standards, even after 20 nodes
and in extreme temperature variations.
Illustration 5.3 Test results over 20 nodes
While Synchronous Ethernet is a Layer 1 enhancement providing
for the relay of timing information completely independent of user
tra"c volume, the remaining technologies to be discussed are all
packet based and are impacted by the user tra"c. This means
that the packets carrying the timing information must compete
for network resources with all of the other data services and the
routing protocol packets. Signicant work has been conducted
over the past several years which has led to implementations of
these technologies being able to meet the required performance
targets over relatively noisy environments where variable queuing
and processing delays can be introduced.
Since changes in network conditions with respect to tra"c load
can a!ect the performance of these technologies, there is the
need to provide more management and monitoring ability for
these services than might be required for Layer 1 technology like
Synchronous Ethernet. Alcatel-Lucent has architected its Timing
Over Packet solutions to include the OAM&P capabilities to allow
these technologies to be deployed with success.
As indicated above, all of these ToP technologies are a!ected by
tra"c load in the network. The key parameter that relates to the
performance is the variation in the Packet Transfer Delay (PTD)
across the network or the Packet Delay Variation (PDV). There
are di!erent Timing over packet technologies such as Adaptive
Clock Recovery (ACR), Di!erential Clock Recovery (DCR),
Network Timing Protocol (NTP) and IEEE1588v2. Because of the
requirements of synchronization accuracy in power utility
networks, we will focus on the IEEE1588v2 standard because
none of the other Timing Over Packet solutions meet the
requirements in terms of number of hops they support and
accuracy hey provide.
IEEE1588v2 and its Precision Time Protocol (PTP) message
exchange is a mechanism that can be used to synchronize time
and timing within a network. Version 1 of this standard is
currently being used in the LAN environment of industrial
manufacturing. It uses a very similar concept of time-stamped
packets between master and slave network elements to NTP but
includes some enhancements such as higher packet rate and
hardware-based time-stamping to improve on the accuracies of
the recovered time. IEEE1588v1 has demonstrated accuracies in
the one microsecond range in the LAN environment. However,
Section 3
Precision Timing Protocol IEEE 1588v2
73
when applied to the noisier WAN environment, it cannot
guarantee this performance. Version 2 was created to try to
address this noisier environment.
Two signicant concepts within IEEE1588v2 are the boundary
clock and the transparent clock. The boundary clock is a device
which has at least one slave port recovering timing/time from an
upstream master and it then uses this recovered timing/time as a
basis for one or more master ports toward downstream slave
ports. The boundary clock can then be used both for scaling
purposes and as an intermediate device to break up the PDV
between the grandmaster and the slave devices. The transparent
clock is a device that participates in IEEE1588v2 but does not
perform any timing/time recovery. The transparent clock
measures the residence time of each PTP message as the
message transits the node and updates the message with this
residence time. Since most of the PDV is caused by queuing
within the nodes, the transparent clock can remove this unknown.
If every device between the master port and the slave port
performs as a transparent clock, then the actual transit time for
each message can be measured and corrected. In an ideal
IEEE1588v2 network, every device along the path from the
Grandmaster clock to the slave clock is IEEE1588v2 aware
and acts as a boundary or transparent clock. However, many
implementations have been developed that can meet the
performance targets in environments where there are a number of
non-IEEE1588v2-aware devices between the master and slave
clocks.
1588v2 denes 5 types of PTP devices.
# Ordinary clock
# Boundary clock
# End-to-end transparent clock
# Peer-to-peer transparent clock
# Management node
74
Illustration 5.4 1588v2 principles
Ordinary Clock
An ordinary clock is a PTP device which has a single PTP port in
a domain, where a domain consists of a logical grouping of
clocks communicating with each other using the PTP protocol.
An ordinary clock can be the source of time into the network
(grandmaster) or recover frequency from the network(slave).
Boundary Clock
A boundary clock is a PTP device with multiple ports in a domain.
It can have one port operating as a slave recovering frequency
from an upstream master port and other ports acting as masters
to redistribute the recovered frequency to downstream slaves.
As the number of hops between a master and slave increases,
typically so does the PDV incurred. Boundary clocks mitigate the
end to end PDV by breaking the PTP packet ow into more
manageable per hop segments.
Transparent Clock
A signicant portion of the end to end PDV between a master and
slave is caused by queuing within the intermediate network
equipment. A transparent clock forwards and modies PTP
packets to compensate for the residence time within these
intermediate devices. The residence time is placed in the
correction eld within the PTP packet header. There are two
types of transparent clock, end-to-end and peer-to-peer.
End-To-End Transparent Clock
An end-to-end transparent clock compensates for the PTP
packet residence time within a device.
Peer-To-Peer Transparent Clock
A peer-to-peer transparent clock compensates for the residence
time and the propagation delay from the upstream PTP clock.
Use of Peer-To-Peer Transparent Clocks requires the use of the
Pdelay messages rather than the Delay messages in all clocks.
Management Node
A management node is a device that communicates with other
PTP devices using PTP management messages. It can be a
standalone device or a logical entity within another PTP device.
Packet Delay Variation
The variation in the time it takes packets to transit a network is
referred to as PDV. PDV is an important factor that impacts all ToP
techniques like ACR, NTP and IEEE1588. If the packet delay
through the packet network is constant, the arrival rate of packets
at the destination node is constant. It will be relatively easy for the
clock recovery module to recreate a stable frequency, phase and
time of day from the delivered packets. If the packet delay varies,
75
then the recovery algorithm is more rigorously exercised. The
algorithm has to deal with both sudden and slow changes in
packet delivery based on reroutes or slowly increasing loads in
the network, as well as possible changes in the master timing due
to reference switches (needs to be followed) and/or drift in the
local oscillator at the core of the local clock (needs to be
removed). Therefore, a clock recovery design must incorporate a
ltering capability to recognize and remove these e!ects. There
are several causes of packet delay variation on a packet network.
Some examples include:
# Random delay variation (e.g., packet arrivals at queuing points)
# Low frequency delay variation (e.g.,day/night tra"c load
patterns)
# Systemati c del ay vari ati on (e.g., store and forward
mechanisms in the underlying transport layer)
# Routing changes and congestion e!ects
The most signicant factors in the PDV are:
# The number of nodes (which contain queuing points)
# The speed of the interfaces (GE vs. FE, for example)
# Technology of the interfaces (Ethernet, DSL, &Wave)
# The packet size, randomness and load of the aggregate tra"c
in the network
In order to know whether a particular ToP implementation will
meet the performance targets in a given network deployment, it is
desirable to characterize the limits on the PDV that the
implementation can support and to also measure the network
against these limits. Unfortunately, ToP implementations as they
exist today cannot always quantify the PDV metric or the specic
limit. There have been several proposals for how to measure the
PDV (peak-to-peak level, TDEV, minTDEV, MAFE, etc.) but since
implementations use di!erent ltering techniques, it is not clear
that one single PDV metric will apply to all solutions. Even within
one implementation there may be factors built into the timing
recovery algorithm which lter the PDV in multiple ways. In these
cases, one single PDV metric would not be su"cient to dene the
algorithms behavior.
When a PDV metric (or set of PDV metrics) is available, the target
deployment network will need to be measured to ensure that the
PDV metrics are not exceeded. It may be di"cult to create all the
network conditions over which the solution must operate, so
some scenarios will have to be estimated.
It will be essential for any deployed solution to provide some
indication of its performance so that it can be monitored to
ensure that performance limits are being met.
76
Leased Line Replacement
An Ethernet network can provide traditional E1/T1 interfaces and
perform circuit emulation across the network. This conguration
could replace the traditional leased E1/T1 services o!ered by
wireline carriers or the E1/T1 services provided in the owned
power utility network. In this deployment, the Circuit Emulation
Service (CES) must meet the same performance characteristics
of the SDH/SONET transport networks. The principle concern is
the transport of the E1/T1 bits from one location to another. This
means that support for service clock transport is required. This
service clock transport support can be provided using either an
ACR or DCR timing method with the CES.
When access to a traditional synchronization distribution network
with BITS devices is available, then the preferred clock recovery
technique is DCR (as it is immune to PDV). If such a distribution
network was not available, then the option is to either use DCR
on the E1/T1 circuits along with a dedicated ToP technique to
distribute the common reference or to use ACR on the individual
E1/T1 circuits.
Rollout of Synchronous Ethernet
SyncE SDH interop
If the network is providing Ethernet ports on a SDH/SONET
backbone, then there is likely to be an option to upgrade only the
Ethernet interface modules from legacy Ethernet to Synchronous
Ethernet capabilities. The SDH/SONET equipment will already
support a clock architecture conformant to the SDH/SONET
requirements and should be able to operate in Hybrid mode
between SDH/SONET synchronization distribution and
Synchronous Ethernet distribution. In this environment, the
Synchronous Ethernet interfaces can be rolled out only where
needed at the network edge. This allows for a phased transition
from the SONET/SDH network over to an all Synchronous
Ethernet backbone while providing immediate support of
Synchronous Ethernet services at the network edge.
SyncE with ToP
The ToP technologies have to be concerned with the PDV
between the timing master and the timing slave points. Using
SyncE as the synchronization distribution technique over the
Section 4
Experience and recommendations
77
core links (and then ToP only over the last mile links) can be a
cost-e!ective deployment option while ensuring performance is
maintained. This scenario is depicted in illustration 5.4.
Since tra"c loading is lower at the network edge than near the
core hub, the smaller number of core links can be upgraded to
Synchronous Ethernet. This avoids the high PDV that can occur
when the links are run with high loading, while at the network
edge ToP can maintain performance when there are fewer links
and lower loading. As there are many more links at the network
edge and usually multiple transmission technologies, this permits
a cost-e!ective and consistent solution over the last mile.
End-to-end Timing Over Packet
As discussed above, the principle concern with the Timing Over
Packet techniques is the control of the PDV. The larger the
network span between the master and slave agents, the larger
the peak-to-peak PDV and the greater the trend to Gaussian
distribution. In all Timing Over Packet implementations there will
be an engineering limit to the network span over which it will
work. There are three methods to address this. The rst is to use
distributed masters where GPS-based masters are placed within
the network to ensure that the span between the master and
slave agents is reduced to an acceptable limit. The second
method is to use boundary clocks within the network between the
primary master and the edge slave devices. The third method,
applicable to IEEE1588v2 deployments only, is to use transparent
clocks within the network. The use of distributed timing masters,
or boundary clocks using a central primary master, are similar
concepts. Before the avai l abi l i ty of network el ements
incorporating boundary clock functionality, the use of distributed
timing masters will be more common. While the distributed GPS-
based masters are operating in plesiochronous mode, the
di!erence in frequencies at the various receivers is so small (0.01
ppb) that there should be no real impact from this mode of
78
Combining PTP for the last mile with SyncE on the main network
provides a viable and cost e!ective solution.
Illustration 5.5 Using PTP in the last mile
operation. Once the networking elements are upgraded to
support boundary clock functionality, the use of one central GPS-
based timing master and then operating through boundary clocks
to reach the network edges will become more common. In this
case, the end-to-end PDV is replaced by several segments of low
PDV, but with the tradeo! of operating several clock recovery
control loops in series. More analysis is needed to determine if
the use of boundary clocks will become more common. More
data needs to be collected on the tradeo!s between using
boundary clocks to reduce the PDV of a single large network
span into multiple master-slave spans. Is there some optimal
number of internodal links that provide enough PDV reduction
while not introducing too many recovery steps? At least for
frequency distribution, preliminary testing has shown some
benets to having boundary clocks in every network element in
order to ensure the PDV is minimized for each master-slave
association. Some of the results of our tests are shown in
illustration 5.5 below. What this tells us is that it is good practice
to combine PTP and Sync E, the former for Time/Phase, the latter
for frequency accuracy.
All Alcatel-Lucent IP/MPLS routers, the
7750SR, 7705 SAR and 7210 SAS products
support both SyncE and 1588v2 to allow for
the most exible and robust synchronization
schemes.
For a number of the 7705 SAR routers, Alcatel-
Lucent is o!ering a GPS module to be built-in,
this helps to address situations on remote sites
where PTP is not possible due to excessive PDV
and SyncE is not available.
These test illustrate how the use of PTP for ToD and Frequency can be
combines with SyncE to achieve the best possible result for Phase and
Frequency accuracy.
Illustration 5.7 PTP test over 13 nodes, including
microwave links and interfering tra"c
79
Illustration 5.6 GPS module for
7705 SAR-H
Synchronization Management
In networks where synchronization is of utmost importance, as is
the case in power utility networks, it is important to be able to
troubleshoot synchronization issues very rapidly.
As Synchronous Ethernet interfaces roll out in a network and are
used for synchronization distribution, the methods and
procedures developed by the SONET/SDH timing experts in
operation of the distribution network will be relevant to the new
environment. New capabilities unlocked by enhancements to the
SSM of Synchronous Ethernet will become available to assist and
expand on the management of the distribution network. These
capabilities can be integrated into dedicated management
systems or incorporated as subsystems within the overall service
management agents of the network. The intention will be to
develop a comprehensive system to allow for the analysis of the
distribution network to look for optimizations and analyze network
80
Time of Day in and Time of Day Out
Sync In and Out
Illustration 5.8 External synchronization inputs and outputs on 7705 SAR-8
scenarios. In the case of the newer concept of ToP, there is an
even greater requirement for management capabilities to control
and monitor the performance. These capabilities start in the clock
recovery slaves that run in the edge devices but also apply to the
management of the timing masters and the delivery paths
between the masters and slaves. The implementations within the
slaves should be capable of indicating some level of condence
in the accuracy and stability of the recovered timing and time of
day. This will have to be based on the stability of the output of the
PDV ltering algorithm. If the algorithm is having trouble
generating a consistent output, then this will be reected in
variations of the recovered frequency. Since this situation will be
impacted by network loading, this needs to be monitored on a
constant basis and indication given when stability drops. These
indications can then be correlated to changes in network
conditions and corrective action can be performed in order to
reduce the variability seen by the slave.
These OAM&P capabilities need to be included in the solutions as
they are provided. A network manager can monitor performance
and run statistical collection continuously, or on demand, to
investigate network characteristics and performance.
Alcatel-Lucent is working together with Chronos and its
SynchWatch product to deliver an end-to-end synchronization
management solution as part of the 5620 Service Aware Manager.
The Sync management solution provides topological map
information about the synchronization tree, either upstream or
downstream. It also shows the sync state of each element in the
network, whether it is in acquiring, holdover or lock status.
It is also possible to get historical information on synchronization
paths which allows users to trace sync path changes back to
routing changes in the network.
The solution also allows sync path audits to be performed to
check for packet loss, excessive PDV etc.
5620 SAM provides information about the synchronization state of each node
in the network. It shows PTP peering alarms, Threshold Crossing Alerts, path
information between slaves, boundaries and masters
Illustration 5.9 Managing Synchronization with 5620 SAM
81
Chapter 6
Conclusion
IP/MPLS has proven to be the right technology to address
the challenges of power utilities as they need to upgrade
their SONET/SDH networks.
This section highlights the most important aspects to take
into consideration.
83
IP/MPLS is the technology of choice for utilities wanting to
implement converged Smart Grid networks.
IP/MPLS o!ers a superior solution to MPLS-TP with no
compromises regarding layer two or layer three services,
multicast support, scaling, integrated cybersecurity and end-to-
end capabilities from access, over aggregation, edge and core
elements of the network. IP/MPLS is capable to resist to multiple
fai l ures, thi s i s of paramount i mportance for cri ti cal
infrastructures such as power grids. IP/MPLS is also topology
agnostic, it can support ring, star and full mesh topologies.
With the right product choices utilities can implement a utility-
grade network that supports the exibility and scalability of IP,
while maintaining the reliability and predictability of traditional
mission-critical networks.
Over recent years, IP/MPLS networks have demonstrated in
numerous power utility networks around the world the ability to
support the most critical and latency-sensitive utility applications,
including Teleprotection. IP/MPLS has also proven to provide
highly secure, reliable and manageable VPN communications
services for multiple applications that support utility operations
and business objectives, while reducing capital and operational
expenses.
It is important to consider bandwidth requirements and latency
as a consequence of packetization, however the real world
examples illustrated in this book show that excellent results can
be achieved.
Synchronization is crucial to enable migration of the TDM
application onto the IP/MPLS network and to enable next
generation teleprotection and phasor measurement applications.
Having the ability to combine SyncE and 1588v2 brings the best
of both worlds to the network in terms of frequency and phase
synchronization.
A very interesting observation made during the numerous tests
was that teleprotection equipment, designed around Ethernet
interfaces performed signicantly better than the equipment
conceived with legacy interfaces. Because the performance
di!erence is an order of magnitude better, Ethernet based
teleprotection equipment allows for new protection algorithms
84
and schemes to be adopted. Last but not least, it is important to
have access to the right management tools which will make the
provisioning of services, element-, alarms- and synchronization
management as well as service performance management simple
and intuitive.
This will in turn contribute to building better, safer and smarter
power grids that are future proof and Alcatel-Lucent is here to
help make this a reality.
Creos 200kV Substation.
Illustration 6.2 Di!erential protection testing
The Substation WAN & LAN solution o!ered by Alcatel-Lucent consists
of ONE technology, based on IP/MPLS,ONE operating system-SROS,
ONE management system -5620 SAM without making compromises.
Illustration 6.1 Summary
Moving a power utility telecom network to IP/MPLS is no longer a technology
issue, it has become an HR issue
- Cory Struth - Falling Apple Solutions, Canada
ATM
Asynchronous Transfer Mode (ATM) is a standard communications protocol that was
designed to carry all types of data, voice and video at high speed. It was dened in the late
1980s. It uses a xed packet length of 53 bytes. ATM technology has been widely
deployed in enterprise and carrier networks since the early 1990s however most of these
networks have been replaced by IP/MPLS networks as of the early 2000s.
Related Glossary Terms
Index
Drag related terms here
Find Term
C37.94
IEEE standard dened as: An optical interface for use between teleprotection and digital
multiplexer equipment that can operate at a data rate of N times 64 kilobit per second
where N = 1, 2... 12 is described. Requirements for both physical connection and the
communications timing are also included.
Related Glossary Terms
Index
Teleprotection
Find Term
CESoPSN
Circuit Emulation Service over Packet Switched Networks. This is a standard mechanism
dened in RFC5086 to encapsulate data from circuit switched interfaces (n x DS0 in E1/T1)
or legacy serial interfaces (RS232, X.21, V.35) into Ethernet packets.
Related Glossary Terms
Index
Drag related terms here
Find Term
Circuit breaker
Device used in electrical systems to interrupt the ow of electricity. These devices are used
to protect electrical power systems from damage caused by faults such as for instance
short circuits.
Related Glossary Terms
Index
Recloser
Find Term
CWDM
Coarse Wave Division Multiplexing is a technology that allows the bandwidth of an optical
cable to be increased by using light at di!erent wavelengths (colors) simultaneously. Each
color represents an independent channel of 1 or 10Gbps.
Related Glossary Terms
Index
Drag related terms here
Find Term
DS0
The smallest entity within the structure of a SONET/SDH network, typically a 64kb timeslot
of an E1 for instance.
Related Glossary Terms
Index
Drag related terms here
Find Term
DSO
Distribution System Operator. This refers to the electric power distribution companies who
operate the medium voltage (below 50kV) and low voltage (1kV and less) electrical network.
Typically the DSOs connect the end users to the electrical network.
Related Glossary Terms
Index
TSO
Find Term
E&M
Analog voice interface used in telephone systems, often to connect older telephone
exhanges. E & M stands for Earth & Magneto but is often said to be Ear & Mouth. It can
use 2 or 4 electrical wires for the audio plus 2 more wires for the signaling.
Related Glossary Terms
Index
Drag related terms here
Find Term
Ethernet
Standard layer 2 protocol used for data communications networks. Initially it was designed
for Local Area Networks (LAN) at 2, 5 or 10 megabits per second on coaxial cable. In the
late 1980s it was competing with other technologies but these have all disappeared. Today,
Ethernet runs on twisted pair and optical bre at speeds of 1, 10, 40 or 100 gigabit per
second. Ethernet has become the standard for enterprise networks and is now becoming a
popular Wide Area Network technology.
Related Glossary Terms
Index
Drag related terms here
Find Term
G.703
ITU-T standard for transmitting data or voice over T1 or E1 communications circuits. This
is typically n x 64k. The physical interface can be a 75Ohm BNC (coax) interface or a 120
Ohm copper interface (RJ48)
Related Glossary Terms
Index
Drag related terms here
Find Term
GOOSE
Generic Object OrientedSubstation Event. This is part of an IEC 61850 dened Generic
Substation Event (GSE) model which denes a fast and reliable way of sending substation
data to multiple devices. GOOSE relies on multicast and broadcast mechanisms.
Related Glossary Terms
Index
Drag related terms here
Find Term
ICT
Information & Communications Technology. ICT is often used to refer to the industry of
information and telecommunications or the information and communications technology
department of a company.
Related Glossary Terms
Index
Drag related terms here
Find Term
IDU
In-door unit. Typically refers to the equipment that needs to be installed inside a building or
a shelter. Often in relation with ODU (out-door unit) as the antenna of a microwave radio
system.
Related Glossary Terms
Index
ODU
Find Term
IEC
International Electrotechnical Commission. Standard conformance organization for
electrical, electronic and all related technologies.
Related Glossary Terms
Index
Drag related terms here
Find Term
IEC 61850
A standard that denes power substation automation systems design.
Related Glossary Terms
Index
Drag related terms here
Find Term
IEEE 1588v2
Precision Timing protocol dened by IEEE in 2009, designed to deliver Time
synchronization using packets.
Related Glossary Terms
Index
PDV, PTP, SyncE, ToP
Find Term
IETF
Internet Engineering Task Force. Standards organization that has dened the IP, and MPLS
standards
Related Glossary Terms
Index
IP, IP/MPLS
Find Term
IP
Internet Protocol (IP) is a connection-less Layer 3 protocol used by routers to relay
datagrams or packets over telecommunications networks. It denes the addressing and
encapsulation mechanism used to exchange data over router networks. IPv4 is the most
widely deployed variant, the more recent IPv6 is gaining ground because of the depletion of
public IPv4 addresses. IP is used in combination with TCP (Transmission Control Protocol)
which provides the connection oriented context to an IP based communication and will
augment the reliability of the IP communication (TCP/IP).
Related Glossary Terms
Index
IETF
Find Term
IP/MPLS
Internet Protocol/Multi Protocol Label Switching is an IETF standard to allow packet
networks to be used on top of existing Ethernet, IP, SONET/SDH and ATM networks and
can carry any type of tra"c on top of it.
Related Glossary Terms
Index
IETF, MPLS, MPLS-TP
Find Term
Jitter
Small rapid variations in a waveform resulting from uctuations in the voltage supply or
mechanical sources or other sources.
Small irregular movement.
In the context of this book jitter refers to the transmission delay variations that can occur
due to several telecoms network conditions.
Related Glossary Terms
Index
PDV
Find Term
LAN
Local Area Network. Most commonly refers to a L2 Ethernet network in a building or
campus.
Related Glossary Terms
Index
Drag related terms here
Find Term
LSP
Label Switched Path
This is a function of MPLS and denes the path along which data is sent across multiple
hops. The path can be set up in a strict manner dening the exact path packets need to
follow between two end points. LSPs can also be set up to be loose, which allows
another control plane protocol to determine the best route.
Related Glossary Terms
Index
MPLS, MPLS-TP
Find Term
MPLS
Multi Protocol Label Switching
Related Glossary Terms
Index
IP/MPLS, LSP, MPLS-TP
Find Term
MPLS-TP
Multi Protocol Label Switching Transport Prole
This variant of MPLS is limited to transport only, has no L3 awareness and does not rely on
a control protocol to setup Label Switched Paths.
Related Glossary Terms
Index
IP/MPLS, LSP, MPLS
Find Term
ODU
Out door unit. Part of a system that is installed outdoor, for instance an antenna or radio of
a microwave transmission system.
Related Glossary Terms
Index
IDU
Find Term
PDH
Plesiochronous Digital Hierarchy or PDH is a predecessor of SDH. It is a Time Division
Multiplexing technology that runs up to 2 megabits per second and provides up to 32
timeslots of 64 kilobit.
Related Glossary Terms
Index
Drag related terms here
Find Term
PDV
Packet Delay Variation
This refers to the di!erence in delay that may occur for packets to traverse the network
between the same endpoints.
Related Glossary Terms
Index
IEEE 1588v2, Jitter, PTP, SyncE, ToP
Find Term
PTP
Precision Timing Protocol.
Refers to any packet based protocol designed to deliver timing with high precision.
Several variants have evolved over the past years and new standards are still in the process
of being developed.
Related Glossary Terms
Index
IEEE 1588v2, PDV, ToP
Find Term
Recloser
A recloser is a circuit breaker which is equipped with a mechanism to close itself
automatically after a fault has been cleared.
Related Glossary Terms
Index
Circuit breaker
Find Term
ROHS
Restriction Of Hazardous Substances. ROHS is a directive of the European Union which
was introduced in 2003 and came into e!ect in 2006. Its purpose is to restrict the use of a
number of chemicals in electric/electronic equipment and appliances of all sorts. The
chemical substances that are banned are: Lead (Pb), Cadmium (Cd), Mercury (Hg),
Hexavalent Chromium (Cr6+), Polybrominated biphenyl (PBB) and Polybrominated diphenyl
ether (PBDE). The latter two were often used as ame retardants in equipment housing and
cable insulation. Lead was mostly used in electronic printed circuit boards for soldering
components.
Related Glossary Terms
Index
Drag related terms here
Find Term
SAToP
Structure Agnostic Transport over Packet. This method of sending TDM data across packet
networks does not take the structure of SDH/SONET or TDM tra"c such as DS0 channels
into consideration. It is a fully transparent transport mechanism.
Related Glossary Terms
Index
Drag related terms here
Find Term
SCADA
Supervisory, Control & Data Acquisition (SCADA) is a term used to dene systems that are
used to monitor and control industrial processes. It often is based on a master-slave type
of communication system to get data from the process and control it in real time.
Related Glossary Terms
Index
Drag related terms here
Find Term
SDH
Synchronous Digital Hierarchy (SDH) is a standard multiplexing protocol used in
telecommunications networks. It is formalized in the ITU standards G.707, G.784 and G.
803. The basic unit of SDH is STM-1 which runs at 155Mb/s, STM-4 runs at 622Mb/S,
STM-16 runs at 1.2Gb/S, STM-64 runs at 10Gb/s and STM-256 runs at 40gb/s
Related Glossary Terms
Index
SONET, TDM
Find Term
SONET
North American variant to SDH.
Related Glossary Terms
Index
SDH, TDM
Find Term
Substation
Part of the electrical network where voltage is transformed from high voltage to medium
voltage, from medium voltage to low voltage or the other way around.
Related Glossary Terms
Index
Drag related terms here
Find Term
SyncE
Synchronous Ethernet is a standard dened in 2008 and implements L1 synchronization on
Ethernet interfaces in a similar way as it is done in SDH/SONET networks. SyncE is to date
the best mechanism to provide accurate frequency synchronization.
Related Glossary Terms
Index
IEEE 1588v2, PDV
Find Term
TDM
Time Division Multiplexing (TDM) is the technology which is used by PDH and SONET/SDH
systems. It divides a xed length frame into a number of equal sized time slots.
These timeslots can contain data or voice tra"c which are transmitted at xed speed and
constant intervals.
Related Glossary Terms
Index
Drag related terms here
Find Term
Teleprotection
A system or device designed to protect high voltage power networks from damage caused
by faults on the power network.
Related Glossary Terms
Index
Drag related terms here
Find Term
ToP
Time over Packet
Refers to any technology used to deliver timing information over packet networks.
IEEE1588v2 is an example of a ToP implementation.
Related Glossary Terms
Index
IEEE 1588v2, PDV, PTP
Find Term
TSO
Transmission Service Operator. This is a company that operates a high voltage network to
carry electricity from the generation to the distribution networks. These are usually
operating at 100kV- 400kV or higher. These companies are mostly regulated by a
government instance and are reduced to mostly one per country in Europe. The reduction
of the number of TSOs it to prevent network instabilities.
Related Glossary Terms
Index
DSO
Find Term
UIC
Union International de Chemins de Fer. The International union of Railways organization,
headquartered in Paris.
Related Glossary Terms
Index
Drag related terms here
Find Term
WAN
Wide Area Network. This refers to the network which connects di!erent remote locations
together.
Related Glossary Terms
Index
Drag related terms here
Find Term
X.21
Serial communications interface standard as dened by the ITU-T in the 1970s. It typically
uses a 15-pin Sub-D connector. It was designed to run at higher speeds than RS232. It is
using balanced (pairs) transmit and receive channels which can scale to 10Mbps although it
is mostly used between 600bps and 64kbps. X.21 was designed to support synchronous
communications.
Related Glossary Terms
Index
Drag related terms here
Find Term