August 14, 2003 Blackout

Summary Based on Interim Report of the United States Canada Power Outage Task Force November 19, 2003

U.S.-Canada Interim Report

Released November 19, 2003 Result of an exhaustive bi-national

investigation

Working groups on electric system, nuclear

plant performance and security Hundreds of professionals on investigation teams performed extensive analysis

Interim report produced by the teams and

accepted by the bi-national Task Force

Overview

Overview of power system and reliability Pre-outage conditions on August 14 Trigger events and start of cascade Wide area cascade Root causes Next steps

Power System Overview

Reliability Overview
Balance generation and demand Balance reactive power supply and demand Monitor flows and observe thermal limits

Observe power and voltage stability limits

Operate for unplanned contingencies Plan, design and maintain a reliable system Prepare for emergencies
Reliably operate the system you have!
5

3 Interconnections / 10 NERC Regions

NERC Control Areas

NERC Reliability Coordinators

Footprints of Reliability Coordinators in Midwest

NERC Immediate Response to Blackout

First hours
Worked closely with NERC Reliability

Coordinators Identified what had tripped and extent of outage Assessed restoration efforts Maintained open line with DOE/FERC Communicated with DHS, White House, and NRC

First days
Assigned project manager Established Steering Group with

industry executive experts Began organizing investigation teams 90+ volunteers + entire NERC staff
10

Investigation Organization Overview

Steering Group Investigation Team Lead
Project Planning and Support Root Cause Analysis Cooper Systems Investigation Process Review Vegetation/ROW Management

U.S Canada Task Force

MAAC/ECAR/NPCC Coordinating Group

NERC & Regional Standards/Procedures & Compliance

Sequence of Events

Transmission System Performance, Protection, Control Maintenance & Damage Generator Performance, Protection, Controls Maintenance & Damage

MAAC Restoration ECAR Data Requests and Management

Operations - Tools, SCADA/EMS Communications Op Planning

Frequency/ACE

NPCC

MEN Study Group

System Modeling and Simulation Analysis

System Planning, Design, & Studies

11

Data Gathering and Analysis

Three fact-finding meetings August 22 September 8-9 October 1-3 Onsite interviews and inspections Secure database of outage information Extensive corroboration of data to

determine facts Analysis by teams of technical experts

12

Root Cause Analysis

Logical structure for investigating complex

problems Identifies changes, conditions, actions, or inactions at each causal step Starts with final event and drills back through each branch of causal tree. Asks why? at each step. Accurate, reliable, defensible understanding of the root causes.

Successfully used to investigate root causes of PJM voltage stability condition in July 1999 and established history in nuclear and defense industries.
13

Root Cause Analysis Phases

16:15 BLACKOUT

16:06
Initial Focus

Sammis Star Star South Canton Hanna Juniper Chamberlin - Harding Pre-Existing Conditions E.g. voltages, wide- area transfers, line and generator outages, etc.

15:05

14

August 14 Conditions Prior to Blackout

Planned outages Cook 2, Davis Besse nuclear plants East Lake 4, and Monroe 1 Transfers high to northeast U.S. + Ontario Not unusually so and not above transfer limits
Critical voltage day Voltages within limits Operators taking action to boost voltages Frequency Typical for a summer day System was within limits prior to 15:05,
15

on both actual and contingency basis

Warm But Not Unusual for August

16

August 14 Imports to Northeast-Central Compared to 6/1 to 8/13/2003

6000 Max Imports Max Exports 4000 Average 14-Aug 2000

Exports

MW

0
0:00 2:00 4:00 6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00

-2000

Imports

-4000

-6000

-8000

16:00

Hour (EDT)

17

Voltages Prior to 15:05 EDT August 14

18

Frequency Typical for Summer Day

19

Blackout was NOT Caused by

Heavy wide-area transfers Low voltages, voltage collapse Lack of IPP voltage/reactive support Frequency anomalies Cinergy outages starting at 12:08 East Lake 5 trip at 13:31
Contributing factor to later events, but not by

itself causal to the blackout

DPL Stuart-Atlanta trip at 14:02 Contributing factor to loss of MISO real-time monitoring, but not electrically significant 20

Outage Sequence of Events Transmission Map Key

ONTARIO

21

East Lake 5 Trip: 1:31:34 PM

ONTARIO

22

East Lake 5 Exciter Failure Causes Trip

23

Stuart Atlanta Trip: 2:02 PM

ONTARIO

24

MISO State Estimator and Reliability Analysis

MISO state estimator and contingency

analysis ineffective from 12:37 to 16:04

State estimator not solving due to missing

information on lines out in Cinergy then DPL Human error in not resetting SE automatic trigger

Using Flowgate Monitoring tool to monitor

conditions on previously identified critical flowgates

25

FirstEnergy Computer Failures

14:14 Alarm logger fails and operators are not aware 14:20 Several remote consoles fail 14:41 EMS server hosting alarm processor and other
No further alarms to FE operators

functions fails to backup 14:54 Backup server fails

EMS continues to function but with very degraded performance

15:08 IT warm reboot of EMS appears to work but alarm

(59 second refresh) FE system data passed normally to others: MISO and AEP AGC function degraded and strip charts flat-lined

process not tested and still in failed condition No contingency analysis of events during the day including loss of East Lake 5 and subsequent line trips
26

Phone Calls to FirstEnergy

FE received calls from MISO, AEP, and PJM

indicating problems on the FE system but did not recognize evolving emergency

14:32 AEP calls regarding trip and reclose of Star-S.

Canton 15:19 AEP calls again confirming Star-S. Canton trip and reclose 15:35 Calls received about spikes seen on system 15:36 MISO calls FE regarding contingency overload on Star-Juniper for loss of Hanna-Juniper 15:45 FE tree trimming crew calls in regarding Hanna-Juniper flashover to a tree PJM called MISO at 15:48 and FE at 15:56 regarding overloads on FE system

27

Chamberlin-Harding (3:05:41)

28

Chamberlin-Harding Indication of Ground Fault Due to Tree Contact as Measured by DFR at Juniper

29

(3:05:41)

Hanna-Juniper (3:32:03)

30

Hanna Juniper Confirmed as Tree Contact at Less than Emergency Ratings of Line

31

Effects of Ambient Conditions on Ratings

32

(3:05:41)

(3:32:03)

Star- S. Canton (3:41:35)

33

Situation after Initial Trips 3:05:41 3:41:35

ONTARIO

34

Canton Central Tidd (3:45:41)

35

138 kV Lines Overload and Cascade Near Akron

Simulated 138 kV Line Loadings

200 180 Dale-W.Can 138 kV W.Ak-PV Q22 138 kV Cham-W.Ak 138 kV E.LimaN.Fin 138 kV CantC Xfmr 100 80 60 40 20 0
W.Ak-PV Q21 138 kV W.Ak 138 kV Bkr Failure E.Lima-N.Lib 138 kV Hard-Chamb 345 kV E.Lima-N.Fin 138 kV Cham-W.Ak 138 kV Dale-W.Can 138 kV

% of Normal Ratings (Amps)

160 140 120

W.Ak-PV Q21 138 kV Babb-W.Ak 138 kV E.LimaN.Lib 138 kV Clov-Torrey 138 kV

Clov-Torrey 138 kV CantC Xfmr Star-S.Cant 345 kV Babb-W.Ak 138 kV Hanna-Jun 345 kV

Outages

36

138 kV Cascade Contributes Further to Overload of Sammis-Star

Dale-W.Canton 138 kV
16:05:55 EDT

W.Akron 138 kV Breaker Chamberlin-W.Akron 138 kV

Sammis-Star

E.Lima-N.Finlay 138 kV Canton Central Transformer

15:51:41 EDT

W.Akron-Pleasant Valley 138 kV

Babb-W.Akron 138 kV E.Lima-New Liberty 138 kV

Cloverdale-Torrey 138 kV

Star-S.Canton 15:41:35 EDT HannaJuniper

15:32:03 EDT

HardingChamberlin 15:05:41 EDT

80

60

40

140

120

% of Normal Ratings

100

20

37

Sammis-Star (4:05:57.5)

38

Sammis-Star Zone 3 Relay Operates on Steady State Overload

39

Actual Loading on Critical Lines

Harding - Chamberlin Hanna - Juniper Star - South Canton Sammis - Star Line Trip Sammis - Star Star - South Canton Line Trip
Hanna - Juniper Line Trip East Lake 5 Trip

1600

1200

Flows (MW)

800

400
Harding - Chamberlin Line Trip

0 12:00 13:00 14:00 Time - EDT 15:00 16:00

40

Actual Voltages Leading to Sammis-Star

370 Harding - Chamberlin 345 kV Line Trip 350 Hanna - Juniper 345 kV Line Trip
Gaps in Data Records

100% Voltage

95% Voltage 330

Voltage (kV)

90% Voltage 310

Star
290

Star - South Canton 345 kV Line Trip Sammis - Star 345 kV Line Trip

Hanna Beaver Perry

270

250 15:00

Time - EDT

16:00

41

Major Path to Cleveland Blocked after Loss of Sammis-Star 4:05:57.5 PM

Remaining Paths

42

345 kV Lines Trip Across Ohio to West

ONTARIO

43

Generation Trips 4:09:08 4:10:27 PM

ONTARIO

44

345 kV Transmission Cascade Moves North into Michigan 4:10:36 4:10:37 PM

45

Northern Ohio and Eastern Michigan Served Only from Ontario after 4:10:37.5 4:10:38.6 PM

46

Power Transfers Shift at 4:10:38.6 PM

47

Eastern Eastern Michigan (Detroit) Unstable Voltage and Frequency Collapse and Pole Slipping
Ontario Michigan Interface Flow and Voltages Beginning 16:10:38

48

Generator Trips to 16:10:38

49

Generator Trips Next 7 Seconds

50

Overloads on PJM NY Ties 4:10:39 PM

51

PJM NY Separating 4:10:44 PM

52

Cleveland Toledo Island 4:10:39 - 4:10:46 PM Cleveland Blacks Out

53

Northeast Completes Separation from Eastern Interconnection 4:10:43 4:10:45 PM

North of Lake Superior

54

Conditions at Niagara Indicate Progressively Worsening Stability Conditions with Prior Events

55

Island Breaks Up: 4:10:46 4:13 PM

56

Frequency in Ontario and New York during Breakup Niagara Generation Stays with Western NY

57

Generator Trips After 16:10:44

58

End of the Cascade

Some Local Load Interrupted

Areas Affected by the Blackout

Service maintained in some area

59

Blackout Root Cause Group 1 FE Situational Awareness

FE did not ensure a reliable system after

contingencies occurred because it did not have an effective contingency analysis capability FE did not have effective procedures to ensure operators were aware of the status of critical monitoring tools FE did not have effective procedures to test monitoring tools after repairs FE did not have additional high level monitoring tools after alarm system failed
60

Blackout Out Root Cause Group 2 Vegetation Management

FE did not adequately manage tree growth

in its transmission rights of way

61

Blackout Cause Group 3 Reliability Coordinator Diagnostics

MISOs state estimator failed due to a data

error. MISOs flowgate monitoring tool didnt have real-time line information to detect growing overloads MISO operators couldnt easily link breaker status to line status to understand changing conditions. PJM and MISO ineffective procedures and wide grid visibility to coordinate problems affecting their common boundaries

62

Responses from Control Areas and Reliability Coordinators Due December 15

63

Near-Term Industry Actions

Voltage support/reactive supply

Reliability communications

Computer failure response & notifications

Emergency action plans and capabilities

Operator training for emergencies

Vegetation management

Next Steps
U.S./Canada Power Outage TF hearings Public hearings to allow comment on report and input on recommendations
December 4 December 5 December 8 Toronto

NERC next steps NERC executive committees December 11 NERC committees meet January 13-14 Continue investigation

Industry technical conference December 10 Philadelphia

Near term analysis and recommendations in support of U.S. Canada Task Force Long term analysis and recommendations for NERC

64