Hybrid OMS in Hurricane Ike Jeff Lane

Presentation Goals About TVEC Dispatch center setup Our typical storm Hurricane IKE response structure Questions

About Trinity Valley

Located 30 miles south east of Dallas Texas 61,000 meters 6,900 miles of line 151 employees SEDC Billing system DisSPatch OMS (using enhanced Crew Management) ESRI/WindMilMap Porche and Twenty First Century Communications IVR Wireless Matrix AVL integrated with DisSPatch

Service Territory

Dispatch Center Layout

Dispatch Center

Typical Major Storm 50 to 75 concurrent outages affecting 5000 to 7000 meters 40 troubleshooting/repair personnel 1 to 2 contract construction and/or ROW crews 2 Dispatchers 1 to 2 Supervisors 1 to 2 Dispatch helpers Restoration time 10 to 15 hours

Typical Dispatch Structure

Incident Commander

Dispatch Assistant

Kaufman District

Athens District

Dispatcher

Crew Coordinator

Dispatcher

Crew Coordinator

20 30 Troubleshooting Personnel

20 30 Troubleshooting Personnel

District line

Hurricane IKE
Approximately 200 concurrent outages (total of 269) 14,000 out at height of storm (total affected 20,141) Damage:
19 broken poles 14 damaged transformers 93 lines down

66 troubleshooting/repair personnel (16 from other coops and contractors) 6 contract construction crews 10 contract ROW crews Had 20 Dispatchers on Saturday, dropped down to 17 on Sunday Restoration time 42 hours

Hurricane IKE Event structure

Incident Command System Event Name: Hurricane Ike Event Date: 9/13/2008
Incident Commander Jeff Lane Public Information Officer Jerry Boze

Safety Officer Danny Belcher

Liason Officer

Operations Section Chief Tony Watson

Support Section Chief Jerry Woolston

Planning Section Chief Tim Craig

Logistics Section Chief Greg Starek

Accounting Section Chief Frank Skube

South Branch Jimmy Splawn

Customer Service Group Shirley Murphree

Staking

HR

West/North Branch Chris Durden

IT Group Jerry Woolston

Engineering

Facilities

East Branch John Jackson

Mapping

Cedar Creek Branch Rodney Wesley Kaufman/ Tawakoni Branch Sam Grimes

Canton Branch Gene

Dispatch Branch Chase Snuffer Jerry Williams

Operations Section
Operations Chief Tony Athens Dispatch Chase Kaufman Dispatch Jerry

South Branch John(14)

West/North Branch Chris(14)

East Branch Jimmy(13)

Montabla Division Lineman Massey Lake Division Lineman Carroll Springs Division Lineman Tucker Division Lineman

Contractor Division J&L George ROW Division Ramirez Tree Service

Cedar Division Lineman Walton Division Lineman Bartons Chapel Division Lineman Wollow Springs Division Lineman

Contractor Division J&L Chad ROW Division Ramirez Tree Service

Canton Division Lineman Antioch Division Lineman New York Division Lineman Coffee Division Lineman

Contractor Division S&H

Office Field

Combine Branch Rodney

Tawkoni Branch Gene

Canton/CCRK Branch Sam

Combine Division Kenneth Marshall Division Lance Scurry Division Lathem Patton Division Jay

ROW Division(5 Crews) Pete Contractor Division J&L Jimmy Kirk

Duck Cove Division Larry B. Glenn Pines Division Bo Enterprise Division Shayde Cobb Switch Division Rudy

ROW Division Kelly Massey

Mabank Division Paul Jackson Division Jamie Tool/Seven Points Division Eddie Kemp Division Virgil Rand Division Floyd

ROW Division(5 Crews) Paul Contractor Division J&L Jason

ROW Division Ramirez Tree Service

Contractor Division J&L Jimmy Ray

Talty Division Raymond

District Dispatchers
Operations Chief Tony

Athens Dispatch Chase

Kaufman Dispatch Jerry

Dispatchers managed the OMS.

Evaluated and corrected predictions when possible Assigned and printed outage tickets to appropriate branches
Outage tickets included Name, Map #, Meter #, Address, Phone #, Outage #, and included all calls received for the outage.

When outages were restored, the tickets were brought back to the Dispatcher to be restored in the OMS.

Branches
Operations Chief Tony Athens Dispatch Chase Kaufman Dispatch Jerry

South Branch John(14)

West/North Branch Chris(14)

East Branch Jimmy(13)

Combine Branch Rodney

Tawkoni Branch Gene

Canton/CCRK Branch Sam

Branch Directors
Received outage tickets from Dispatcher and dispatched to appropriate Division leaders Worked with Dispatcher to modify any incorrect outage predictions Coordinated Construction/ROW contractor movement between Division Leaders within their branch Issued line clearances to Division Leaders Record restoration date/time, cause, and equipment codes on outage tickets once restored Return outage tickets to Dispatcher for closing in the OMS

Divisions
Operations Chief Tony Athens Dispatch Chase Kaufman Dispatch Jerry South Branch John(14) West/North Branch Chris(14) East Branch Jimmy(13) Combine Branch Rodney Tawkoni Branch Gene Canton/CCRK Branch Sam

Montabla Division Lineman Massey Lake Division Lineman Carroll Springs Division Lineman Tucker Division Lineman

Contractor Division J&L George ROW Division Ramirez Tree Service

Cedar Division Lineman Walton Division Lineman Bartons Chapel Division Lineman Wollow Springs Division Lineman

Contractor Division J&L Chad ROW Division Ramirez Tree Service

Canton Division Lineman Antioch Division Lineman New York Division Lineman Coffee Division Lineman

Contractor Division S&H

Combine Division Kenneth Marshall Division Lance Scurry Division Lathem Patton Division Jay Talty Division Raymond

ROW Division(5 Crews) Pete Contractor Division J&L Jimmy Kirk

Duck Cove Division Larry B. Glenn Pines Division Bo Enterprise Division Shayde Cobb Switch Division Rudy

ROW Division Kelly Massey

Mabank Division Paul Jackson Division Jamie Tool/Seven Points Division Eddie Kemp Division Virgil Rand Division Floyd

ROW Division(5 Crews) Paul Contractor Division J&L Jason

ROW Division Ramirez Tree Service

Contractor Division J&L Jimmy Ray

Division Leaders
Received outage information from Branch Directors via cell phone and radio Coordinated personnel with their divisions to restore power Communicated resource needs back to Branch Director (i.e., need more or less crews) Communicated restoration progress back to Branch Director

How did it actually work???

After the event was over, we conducted post-event reviews with everyone involved, from the CEO to field personnel. All of the feedback we received was positive. Typical bottlenecks at the Dispatch level were avoided by dividing the workload into manageable levels. Even the linemen who were acting as Division Leaders and performing dispatch functions for the personnel under them felt like we should continue to use this structure in future events. The normal Operations department supervisors were asking to begin using this structure more often (i.e., events that may not be as large as a hurricane but that are larger than a typical major thunderstorm). The normal Dispatchers loved it because they had a manageable workload. Everyone felt like this provided a safer working environment because they werent being overworked and that reduced the possibility for mistakes. The general consensus was that restoration was achieved sooner than it would have been if we had used the normal Dispatch structure.

Did we have issues???

Sure we did. Some of the division leaders were holding onto restoration information until they had several outages restored. Their reasoning was they wanted to report them all in one call back to the Branch Director instead of multiple calls. This caused issues in the office with accurate reporting to the PUC, Critical Account restoration status, and planning of the next days work. Outage re-predictions were a problem. The Dispatcher would give the Branch Director several outages, all on separate outage tickets, then a reprediction would push those outages upline. We had to manage that paper with effective communications between the Dispatchers and the Branch Directors. Reallocating workload dynamically based on restoration status was challenging but worked with effective communication between Branch Directors and the Operations Section Chief.

Will this structure work to manage The Big One?

With minor modifications, I believe it will. Thats just one opinion, though, and I havent been through The Big One. What is your opinion?