The Devil is in the Details

Lessons in the Design and Specification

of Reliable Standby Power Systems
December 16, 2015

Topics to Be Covered

Standby Power Reliability Data

Real-World Failures

Case 1: Engine Cooling System

Case 2: Engine Starting Batteries

Case 3: Circuit Breaker Failure

Case 4: Protective Relay Trip

Tips for the Design and Specification

Process

Standby Power Reliability Data

Overall Failure Rates (IEEE Standard 493)

Packaged Units: 0.1235 failures per year

Unpackaged Units: 0.63299 failures per year

If the average unit runs 20 hours per year for testing and
outages this is one failure every 161 hours of operation for
packaged units and one every 31 hours of operation for
unpackaged units.

Corresponding 1-year reliability:

Packaged Units: 88.3%

Unpackaged Units: 53.1%

Standby Power Reliability Data

Nuclear Power Industry Data

INEL 1996 Report

Failure rates (per hour)

0-1/2 Hour:

.025

Hour 14 Hours:

.0018

Over 14 Hours:

.00025

Overall reliability (per

demand): > 95%

Most Common Failure Causes

Failure to Start

ELECTRICAL

FUEL

AIR START

0%

20%

40%

60%

80%

100%

Most Common Failure Causes

Failure to Run
FUEL

ELECTRICAL

COOLING

ENGINE MECHANICAL

LUBRICATION

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Case 1: Generator Cooling System

REMOTE
RADIATOR

REMOTE
RADIATOR

REMOTE
RADIATOR

DIESEL
ENGINE

DIESEL
ENGINE

DIESEL
ENGINE

Case 1 Improved System

REMOTE
RADIATOR

DIESEL
ENGINE

REMOTE
RADIATOR

DIESEL
ENGINE

REMOTE
RADIATOR

DIESEL
ENGINE

How Improved Is It?

For 3 independent engine generator / radiator
combinations without common headers:
Assume the probability of failure P(f) of an
engine generator is 10% and P(f) of a remote
radiator is 5%
Then for a single engine generator with a
dedicated remote radiator P(f) = 5% + 10% =
15%, or 0.15
For an N+1 system, two units must fail to lose
the load and there are three combinations in
which two units can fail, thus P(f) = 3 x (0.15 x
0.15) = .0675 or 6.75%
9

How Improved Is It?

For 3 engine generators and 3 radiators with
common supply and return headers:

The probability of two of three radiators

failing is 3 x (0.05 x 0.05) = 0.0075

The probability of two of three generators

failing is 3 x (0.10 x 0.10) = 0.03

Since either of these conditions will cause loss

of load, the probability of one or the other or
both occurring is their sum: P(f) = 0.03 + 0.075
= 0.0375 or 3.75%

Common headers reduce the calculated failure

probability by a factor of 2

10

What Happened

One of two utility services to the hospital

experienced an outage

Automatic transfer controls on the normal

power system operated correctly to restore
power to all loads

While the utility was working to restore the

failed service fuses on the remaining service
transformer blew

All three generators started and paralleled to

the emergency bus, restoring service to the
essential power system

11

Then

A flexible connector on one generator failed,

resulting in a major coolant leak

All three generators shut down on low coolant

level

Operations staff were able to isolate the units,

refill one engine and radiator by hand, and get
that generator running, providing limited
essential power

Critical care patients were evacuated to

another hospital and the event made the
newspapers

12

Potential Single Failure Points

Engine room ventilation controlled by BAS

Fuel Systems

13

Transfer pumps from bulk storage tank

Anti-siphon valves

Control Systems

Power supplies

Communication networks and switches

Best battery systems

Case 2: Engine Starting Batteries

Starting batteries were determined to have

inadequate capacity to obtain the coderequired 10-second start of engine-generator

Gen-set distributor replaced the original

batteries with larger capacity units

14

What Happened:

When the batteries were disconnected, a

system alarm was annunciated indicating loss
of communications with the generator PLC

This was expected, as the generator PLCs

were supplied from the starting batteries and
the alarm was acknowledged

When the work was completed, the generator

PLC alarm failed to clear

The PLC was determined to have a power

supply failure

15

Problem: Battery Charger Specification

A Battery charger consists of a full-wave rectifier
with (or without) filtering that converts a 60 HZ AC
sine wave of voltage to a DC voltage

UNFILTERED RECTIFIER OUTPUT

16

FILTERED RECTIFIER OUTPUT

Chargers for batteries serving control systems

should be specified with filtering; also called
battery eliminator feature

Case 3: Circuit Breaker Failure

FROM UTILITY

TO STEP-DOWN
TRANSFORMERS
SERVING
NORMAL LOADS

17

TN

TE

DG

DG

G1

G2

F1

F2

TO STEP-DOWN
TRANSFORMERS SERVING
EMERGENCY AND STANDBY
LOADS

What Happened

18

System was tested by opening breaker TN to

simulate loss of utility

Breaker TE opened, generators started and

paralleled to emergency bus

Breaker TN was closed to simulate return of

utility power.

Generators synchronized to utility, breaker TE

closed, breakers G1 and G2 opened and
generators shut down

At the next system test, when breaker TN was

opened breaker TE failed to open.

Typical Breaker Control Schematic

19

Potential Solutions

Test more often

Install control relays to monitor voltage

downstream of fuses and input to system
controls as an alarm

Monitor alarm contact or power supply failure

contact on protective relays

Use discrete inputs on protective relays to

monitor control circuit voltages

20

Case 4: Protective Relay Trip

21

2000 kW diesel generator serving multiple

750kVA to 1000 kVA 13.8 kV-480/277 V building
transformers tripped on generator differential
protection when emergency feeder energized

This is a single unit serving a dedicated block

of load without redundancy, so effect was
complete loss of standby power to affected
buildings

Determined post-event that there was no fault

in the generator

Differential Protection

FAULT

22

Differential Protection

FAULT

23

Cause of the Problem

24

The current transformers (CTs) on the bus

side and the neutral side had the same
ratio, but different characteristics and
accuracies
Under the high inrush current associating
with energizing the transformers, one set of
CTs saturated before the other, leading to a
difference in their output currents which
the relay interpreted as a fault in the
generator zone
The fix: replace the relay with a newer
version that has saturation detection logic
and de-sensitize the differential setting

Beware of Protective Relaying

25

Modern microprocessor-based relays are both

a blessing and a curse

They are a blessing because:

Sophisticated protection is available at low cost

and with minimal panel space required

Integral metering and communication capabilities

further reduce overall cost and can improve
reliability by reducing required component count

On-board diagnostic tools including oscillography

and event recording support troubleshooting

Multifunction vs. Discrete Relays

9 discrete relays with

unique protective
functions replaced by
a single uP relay

26

Typical Multifunction GPR

27

Beware of Protective Relaying

28

They are a curse because:

Sophisticated protection elements designed for

large central station generators may mis-operate
for transient conditions experienced on standby
systems

They can be complicated to properly program

subtle differences between manufacturers and
long learning curves contribute to errors in settings

The result: Statistically, nuisance tripping

occurs with a frequency 15-20 times that of
failure to trip
The good news: The powerful diagnostics
make it much easier to figure out what you
did wrong!

Design Suggestions

Keep as much scope as possible and provide

detailed design in lieu of performance
specifications

Pay as much attention to the design of controls

and auxiliary systems as to the basic
configuration of the standby power system

Failure Modes and Effects Analysis (What If?)

Benefits:

29

Quality

Control

Documentation

Specification Suggestions

Require the design of auxiliary and control

systems to provide the same level of
redundancy and reliability as the system
configuration

Designate a responsible entity to integrate

controls and consolidate information from
multiple equipment suppliers into a single set
of drawings

Require complete and detailed submittals of

schematics and wiring diagrams

Review those submittals in detail

30

Questions?

31