Boiler Reliabiliti Optimization 2001

EPRI Boiler Reliability Optimization
Program
Case Studies from 1998-2001
SED
R I
A L
LICE
M AT E
WARNING:
Please read the License Agreement
on the back cover before removing
the Wrapping Material.
Technical Report
EPRI Boiler Reliability Optimization

Program
Case Studies from 1998-2001
1006537
Final Report, December 2001
EPRI Project Manager

P. Abbott
EPRI 3412 Hillview Avenue, Palo Alto, California 94304 PO Box 10412, Palo Alto, California 94303 USA
800.313.3774 650.855.2121 askepri@epri.com www.epri.com
DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES

THIS DOCUMENT WAS PREPARED BY THE ORGANIZATION(S) NAMED BELOW AS AN
ACCOUNT OF WORK SPONSORED OR COSPONSORED BY THE ELECTRIC POWER RESEARCH
INSTITUTE, INC. (EPRI). NEITHER EPRI, ANY MEMBER OF EPRI, ANY COSPONSOR, THE
ORGANIZATION(S) BELOW, NOR ANY PERSON ACTING ON BEHALF OF ANY OF THEM:
(A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I)
WITH RESPECT TO THE USE OF ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR
SIMILAR ITEM DISCLOSED IN THIS DOCUMENT, INCLUDING MERCHANTABILITY AND FITNESS
FOR A PARTICULAR PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR
INTERFERE WITH PRIVATELY OWNED RIGHTS, INCLUDING ANY PARTY'S INTELLECTUAL
PROPERTY, OR (III) THAT THIS DOCUMENT IS SUITABLE TO ANY PARTICULAR USER'S
CIRCUMSTANCE; OR
(B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER
(INCLUDING ANY CONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE
HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES) RESULTING FROM YOUR
SELECTION OR USE OF THIS DOCUMENT OR ANY INFORMATION, APPARATUS, METHOD,
PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT.
ORGANIZATION(S) THAT PREPARED THIS DOCUMENT
Eskom
ORDERING INFORMATION
Requests for copies of this report should be directed to EPRI Customer Fulfillment, 1355 Willow Way,
Suite 278, Concord, CA 94520, (800) 313-3774, press 2.
Electric Power Research Institute and EPRI are registered service marks of the Electric Power
Research Institute, Inc. EPRI. ELECTRIFY THE WORLD is a service mark of the Electric Power
Research Institute, Inc.
Copyright 2001 Electric Power Research Institute, Inc. All rights reserved.
CITATIONS
This report was prepared by
Eskom
Megawatt Park
P.O. Box 1091
Johannesburg, 2000
South Africa
Principal Investigator
D. McGhee
This report describes research sponsored by EPRI.
The report is a corporate document that should be cited in the literature in the following manner:
Boiler Reliability Optimization Project Case Studies from 1998 2001, EPRI, Palo Alto, CA:
2001. 1006537.
iii
REPORT SUMMARY
The prospect of competing in a deregulated environment is driving utilities to explore more costeffective means of operating and maintaining their plants. Recognizing this, a number of
companies including American Electric Power, Great River Energy, Detroit Edison, Arizona
Public Services Company, and Hawaiian Electric Company have participated in EPRIs Boiler
Reliability Optimization Program. This report provides seven case studies demonstrating key
aspects of the program.
Background
EPRIs Boiler Reliability Optimization Program consists of several customized projects. Each
project includes four phases that involve EPRI technical staff, with participation from the
plants operating, maintenance, and engineering support staff, as follows: Phase 1technical
and organizational assessments, Phase 2development of a boiler failure defense plan, Phase
3implementation of a failure defense plan, and Phase 4continuous improvement and process
automation. Each phase, in turn, builds on the strengths of the preceding phase. EPRI compiled
these case studies to demonstrate the scope of the program and its applicability to a wide range
of power production issues.
Objective
To provide case studies from membership participation in EPRIs Boiler Reliability
Optimization Program between 1998 and 2001.
Approach
The principal investigators on this report selected case studies that would demonstrate key facets
of the program, including development of a maintenance strategy, implementation of
Streamlined Reliability Centered Maintenance (RCM) procedures, outage task prioritization,
Boiler Predictive Maintenance (BPdM), performance of root cause analysis, and application of
EPRIs Boiler Maintenance Workstation (BMW) software. Each case study includes a scope of
work description, findings and observations, conclusions, and recommendations.
Results
Two case studies from American Electric PowerBig Sandy Plant-Unit 2 demonstrate phases 1,
2, and 4 of the program, with specific application to water- and steam-touched tubing. This
project focused on the long-term integrity of the water- and steam-touched tubing within the
boiler envelope. It involved training on EPRIs Streamlined RCM software using the fuel system
as an example, additional training on EPRIs tube failure reduction program, development of a
BPdM process, and the provision of EPRIs BMW software package for trending and tracking
tube failure data and information. An outage task prioritization modelRisk Evaluation and
Prioritization (REAP)was used to prioritize the tasks identified for the Unit 2 scheduled outage
in April 2000.
A case study from Great River Energys Coal Creek Station-Unit 2 discusses the decision to
change the boiler maintenance overhaul frequency from two to three years. The project
emphasizes development of a comprehensive and effective boiler inspection plan to take the
place of the currently reactive plan, which focuses almost entirely on sootblower erosion and
does not cover all boiler components in sufficient detail.
A case study from Detroit Edisons St Clair Plant-Unit 7 focuses on technical and programmatic
opportunities for improving boiler reliability. The project identifies the causes of boiler tube
failures and highlights a number of opportunities for improvement with respect to managing the
short- and long-term health of the boiler.
Two case studies at Arizona Public Services Companys Four Corners and Cholla plants
emphasize root cause analysis. A review of the 1999 Lost Generation Action Plans from both
plants evaluates whether the true root causes of lost generation have been identified, and
whether appropriate action plans are in place to address these causes.
A case study from Hawaiian Electric Company (HECO) addresses EPRIs independent review of
HECOs Remaining Useful Life and Generation Asset Management Program. The EPRI team
identified a number of opportunities where current project management and root cause analysis
efforts could be enhanced to provide additional assurance that availability and reliability goals
would be achieved.
EPRI Perspective
A Boiler Reliability Optimization Program is only as effective as the people, management
systems, component manufacturers, technology specialists, and predictive maintenance tools or
equipment that comprise it. For a condition-based, highly planned, maintenance optimization
program to be effective, it must integrate the work process, management, work culture,
technologies, and people into the total plant operation. EPRI believes this documentation of case
studies demonstrates innovative applications of EPRIs Boiler Reliability Optimization Program
and methods for achieving such integration under a wide variety of circumstances. Utilities will
find that implementation of the Boiler Reliability Optimization Program will help them compete
cost-effectively in the deregulated power generation market.
Keywords
Boiler reliability
Boiler Reliability Optimization Program
Competitive power generation
vi
EPRI Licensed Material
CONTENTS
1 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1, 2 & 4 AT AMERICAN

ELECTRIC POWERS BIG SANDY PLANT UNIT 2............................................................ 1-1
1.1
EXECUTIVE SUMMARY........................................................................................... 1-1
1.2
INTRODUCTION ...................................................................................................... 1-2
1.3
SCOPE OF WORK ................................................................................................... 1-3
1.4
FINDINGS AND OBSERVATIONS............................................................................ 1-3
1.4.1 Phase I and II - Technical and Organisational Assessment.................................. 1-3

1.4.1.1 Work Process................................................................................................ 1-5
Maintenance Strategy........................................................................................... 1-7
Findings:............................................................................................................... 1-8
Recommendations:............................................................................................... 1-8
Work Identification ................................................................................................ 1-8
Findings:............................................................................................................... 1-9
Recommendations................................................................................................ 1-9
Work Control .......................................................................................................1-10
Findings...............................................................................................................1-10
Recommendations...............................................................................................1-10
Work Execution ...................................................................................................1-11
Findings...............................................................................................................1-11
1.4.1.2 Technologies................................................................................................1-11
Findings...............................................................................................................1-12
Recommendation ................................................................................................1-12
Work Management System .................................................................................1-12
Findings...............................................................................................................1-12
Maintenance & Diagnostic Technologies .............................................................1-12
Findings...............................................................................................................1-12
vii
Information Integration.........................................................................................1-13
Findings...............................................................................................................1-13
1.4.1.3 Management and Work Culture....................................................................1-13
1.4.1.4 People..........................................................................................................1-15
Findings:..............................................................................................................1-15
1.4.2 Failure Defence Plan Water and Steam Touched Tubing..................................1-15
Findings ...................................................................................................................1-16
Economiser .........................................................................................................1-16
Waterwalls...........................................................................................................1-16
Platen Superheat.................................................................................................1-17
Finishing Superheat.............................................................................................1-17
First Reheat (High Pressure) ...............................................................................1-17
Second reheat (low pressure)..............................................................................1-18
Spray Attemperators............................................................................................1-18
1.4.3 Streamlined Reliability-Centered Maintenance.....................................................1-30
1.4.4 Unit 2 Outage Task Prioritisation..........................................................................1-30
1.4.5 Boiler Tube Failure Reduction and Cycle Chemistry Improvement Training .........1-33
1.4.6 Boiler Maintenance Workstation (BMW) installation and training..........................1-33
AEP Big Sandy Unit 2, Boiler Reliability Optimization - Phase 4, July 2001.......................1-34
Executive Summary ..........................................................................................................1-34
Introduction .......................................................................................................................1-34
Description of Phase 4 of a Typical Boiler Reliability Optimization Project.........................1-35
Findings ........................................................................................................................1-35
Conclusions.......................................................................................................................1-40
Recommendations ............................................................................................................1-41
Appendix 1: Interview List..................................................................................................1-44
Appendix 2: List of Typical headings that appear in a policy and or a procedure ...............1-45
viii
2 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 2 BOILER

INSPECTION PLAN REVIEW AT GREAT RIVER ENERGYS COAL CREEK STATION
UNIT 2 ................................................................................................................................. 2-1
2.1 Executive Summary..................................................................................................... 2-1
2.2 Introduction.................................................................................................................. 2-1
2.3 Findings and Observations .......................................................................................... 2-2
2.4 Recommendations....................................................................................................... 2-3
ANNEXURE I: Extract from a Boiler Inspection Report....................................................... 2-7
Inspection Areas ....................................................................................................... 2-7
1200 Waterwall Cleaning Devices, operational condition ............................................... 2-7
Inspection results........................................................................................................... 2-7
Recommendations for 2004: .......................................................................................... 2-8
1501 Superheat Division Panels, attachment problems ................................................ 2-8
Inspection results:.......................................................................................................... 2-8
Recommendations for 2004: .......................................................................................... 2-8
ANNEXURE II: Unit 2 Outage Inspection Plan as of August 21, 2000 ................................ 2-9
1. Burner Front Team Member...................................................................................... 2-9
1.1 Prior to the outage (within 2 weeks of the start of the outage) ............................. 2-9
1.2 At the start of the outage (During the installation of the boiler scaffolding)..........2-10
1.3 Burner Front inspection (Upon completion of the boiler scaffolding) ...................2-10
2. Waterwall Team Member .........................................................................................2-12
2.1 Prior to the outage (within 2 weeks of start of the outage) ..................................2-12
2.2 Prior to the installation of the boiler scaffolding (after the throat scaffolding is
installed) ..................................................................................................................2-12
2.3 Upon completion of the boiler scaffold................................................................2-12
2.4 Upon completion of the bottom ash scaffolding ..................................................2-13
3. Superheat and Reheat Team Member .....................................................................2-14
3.2 During the outage...............................................................................................2-14
4. Back-pass Team Member ........................................................................................2-16
5. External Boiler Team Member .................................................................................2-18
5.1 Prior to the outage (within 2 weeks of the start of the outage) ............................2-18
ix
3 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1 AT DETROIT

EDISONS ST CLAIR PLANT UNIT 7 AMERICAN ELECTRIC POWERS BIG
SANDY PLANT UNIT 2 ....................................................................................................... 3-1
Executive Summary ........................................................................................................... 3-1
3.1 Introduction.................................................................................................................. 3-3
3.2 Findings and Observations .......................................................................................... 3-3
3.2.1 Programmatic and Data Sources Assessment ..................................................... 3-3
3.2.1.1 Maintenance Strategy................................................................................... 3-6
3.2.1.1.1 Findings ................................................................................................ 3-6
3.2.1.1.2 Recommendations ................................................................................ 3-7
3.2.1.2 Work Process............................................................................................... 3-7
3.2.1.2.1 Findings ................................................................................................ 3-8
3.2.1.2.2 Recommendations ................................................................................ 3-8
3.2.1.3 Predictive Maintenance Program and Technology.......................................3-10
3.2.1.3.1 Findings ...............................................................................................3-10
3.2.1.3.2 Recommendations ...............................................................................3-11
3.2.1.4 People.........................................................................................................3-11
3.2.1.4.1 Findings ...............................................................................................3-11
3.2.1.4.2 Recommendations ...............................................................................3-12
3.2.2 Failures and Root Causes...................................................................................3-12
3.2.2.1 Boiler Tubes ................................................................................................3-12
3.2.2.1.1 Failure Mechanisms, Actions taken and Recommendations ................3-14
3.2.2.2 Boiler Headers.............................................................................................3-18
3.2.2.2.1 Boiler Test and Inspection Plan Guideline............................................3-19
Purpose...............................................................................................................3-19
Definitions............................................................................................................3-19
Scope ..................................................................................................................3-19
Inspection and Test Plan Development ...............................................................3-20
3.3 Conclusions................................................................................................................3-21
3.4 Recommendations......................................................................................................3-21
Appendix 1: List of interviewees ........................................................................................3-23
Appendix 2: Example of a Typical Equipment Condition and Technology Matrix ...............3-24
Appendix 3: Remnant Life Assessment Flow Chart for Thick Section................................3-25
4 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1- ROOT CAUSE

ANALYSIS AT ARIZONA PUBLIC SERVICES COMPANYS FOUR CORNERS AND
CHOLLA PLANTS.................................................................................................................. 4-1
FOUR CORNERS PLANT .................................................................................................. 4-1
4.1 Executive Summary...................................................................................................... 4-1
4.2 Introduction................................................................................................................... 4-2
4.3 Plant Description .......................................................................................................... 4-3
4.4 Incidents investigated ................................................................................................... 4-3
4.4.1 Unit 5 #4 Turbine Control Valve chatter - 197,471 Mwh lost................................ 4-4
Incident Description................................................................................................... 4-4
Root Cause Analysis Chart ....................................................................................... 4-4
4.4.2 Unit 3 Scrubber unit failure - 33,446 Mwh lost...................................................... 4-5
Root Cause Analysis Charts...................................................................................... 4-6
4.4.3 Unit 4 Loss of Booster Fans
38,413 Mwh.......................................................... 4-7

4.4.4 Unit 3 Pulverizer Pinion & Bull Gear Failure - 34,881 MW Hrs. ............................ 4-7
4.4.5 Unit 3 Back-pass Flyash Erosion 39,378 Mwh ................................................... 4-8
Problem Description.................................................................................................. 4-8
4.4.6 Unit 4 Air Pre-heater Failure 166,744 Mwh. ....................................................... 4-9
4.4.7 Units 4 Flyash Erosion 39,288 MW/Hrs.............................................................. 4-9
Root Cause Analysis Chart ......................................................................................4-10
4.4.8 Unit 3 Turbine H.P. Nozzle Pluggage 100,628 Mwh..........................................4-10
Problem Description.................................................................................................4-10
4.4.9 Unit 4 LP Exciter Field Failure 45,229 Mwh........................................................4-11
Incident Description..................................................................................................4-11
Root Cause Analysis................................................................................................4-11
xi
4.4.10 Unit 4 HP Exciter Field Failure 33,500 Mwh .....................................................4-11

Root Cause Analysis................................................................................................4-11
4.5
Findings ...................................................................................................................4-12
4.6 Conclusions.................................................................................................................4-12
4.7 Recommendations.......................................................................................................4-13
4.8 Appendix 1: Root Cause Analysis Charts ...................................................................4-13
CHOLLA PLANT ...............................................................................................................4-26
4.9 Executive Summary.....................................................................................................4-26
4.10 Introduction................................................................................................................4-27
4.11 Plant Description .......................................................................................................4-28
4.12 Incidents Investigated................................................................................................4-28
4.12.1 Mechanical Dust Collector Retrofit 41,146 Mwh. ..............................................4-29
4.12.2 Unit One ID Booster Fan Bearing Failures 7,544 Mwh. ....................................4-30
Root cause Analysis Chart .......................................................................................4-30
4.12.3 Unit one Turbine Thrust Trips 9,971 Mwh. .......................................................4-31
4.12.4 Unit Two A Booster ID Fan Motor Failure 30,464 Mwh. ................................4-32
4.12.5 Unit Four Air Pre-heater Pluggage 48,740 Mwh. ..............................................4-33
4.12.6 Unit Four Scrubber Problems 76,917 Mwh.......................................................4-34
4.12.7 Unit Four High Opacity Shutdown 42,731 ........................................................4-35
4.12.8 Unit Two Waterwall Tubes Hydrogen Damage 188,944 Mwh...........................4-35
xii
4.12.9 Unit Two Cable Tray Fire 28,899 Mwh. .............................................................4-36
4.12.10 Unit Four Air Pre-heater Guide Bearing Fire 6,099 Mwh. ...............................4-38
Root cause Analysis Chart .......................................................................................4-38
4.13
Findings...............................................................................................................4-39
4.14
Conclusions.........................................................................................................4-40
4.15
4.16 Appendix 2 ...............................................................................................................4-40

Root Cause Analysis Charts (Excel file)........................................................................4-40
5 BOILER RELIABILITY OPTIMIZATION PROJECT PROGRAM AUDIT AT
HAWAIIAN ELECTRIC COMPANY........................................................................................ 5-1
5.1 EXECUTIVE SUMMARY ............................................................................................. 5-1
5.2 Introduction.................................................................................................................. 5-1
5.3 Findings....................................................................................................................... 5-2
5.3.1 Boiler Reliability Optimization Project Plan........................................................... 5-2
5.3.2 Project Activities................................................................................................... 5-2
5.3.2.1 BTFR/CCI Program ...................................................................................... 5-2
5.3.2.2 Streamlined Reliability Centered Maintenance ............................................. 5-3
5.3.2.3 Root Cause Analysis .................................................................................... 5-3
5.3.3 Condition based data collection ........................................................................... 5-4
5.4 Conclusions................................................................................................................. 5-4
5.5 RECOMMENDATIONS ............................................................................................... 5-4
Appendix 1: Boiler Reliability Optimization Storage Plan .................................................... 5-7
Appendix 2 ........................................................................................................................5-10
Appendix 3 ........................................................................................................................5-11
List of Typical Headings that Appear in a Policy or a Procedure ...................................5-11
xiii
LIST OF FIGURES
Figure 1-1 Maintenance Optimization Program ....................................................................... 1-4
Figure 1-2 Big Sandy benchmarked against Industry Best Practice ........................................ 1-5
Figure 1-3 Work Process ........................................................................................................ 1-6
Figure 1-4 Work Process Model.............................................................................................. 1-7
Figure 1-5 Task Scatter Diagram - Value to Cost ...................................................................1-31
Figure 1-6 Accumulated Value verse Cost .............................................................................1-31
Figure 3-1 Spider Chart........................................................................................................... 3-4
Figure 3-2 ..............................................................................................................................3-13
Figure 4-1 Unit 5, #4 Control Valve 197,471 Mwh ..................................................................4-14
Figure 4-2 Unit 3 Scrubber Problems 33,446MW/Hrs.............................................................4-15
Figure 4-3 Unit 3, Scrubber Problems 33,446MW/Hrs............................................................4-16
Figure 4-4 Unit 3 Scrubber Recycle Pump Problems 33,446MW/Hrs.....................................4-17
Figure 4-5 Unit 4, Loss of Booster Fans, 38, 413 MW/Hrs .....................................................4-18
Figure 4-6 Unit 3 Pulverizer Pinion and Bull Gear Failures, 38,881 MW/Hrs. .........................4-19
Figure 4-7 Unit 3 Backpass Flyash Erosion 39,378 MW/Hrs. .................................................4-20
Figure 4-8 Unit 4 Air Preheater Failure 166,744 MW/Hrs. ......................................................4-21
Figure 4-9 Units 4 and 5 Backpass Flyash Erosion ................................................................4-22
Figure 4-10 Unit 3 HP Turbine Nozzle Plugging, 100,628 MW/Hrs.........................................4-23
Figure 4-11 Unit One, Mechanical Dust Collection Retrofit, 41,146 Mwhs..............................4-41
Figure 4-12 Unit One, ID Booster Fan Bearing Failures, 7,544 Mwhs ....................................4-42
Figure 4-13 Unit One Turbine Thrust Trips, 9,971 Mwhs........................................................4-43
Figure 4-14 Unit Two, A Booster ID Fan Motor Failure, 30,464 Mwhs....................................4-44
Figure 4-15 Unit Four, Air Preheater Pluggage, 48,740 Mwh .................................................4-45
Figure 4-16 Unit Four Scrubber Outage, 76,917 Mwh ............................................................4-46
Figure 4-17 Unit Four, High Capacity Shutdown, 42,731 Mwh ...............................................4-47
Figure 4-18 Unit Two Waterwall Tubes Hydrogen Damage, 188,944 Mwh.............................4-48
Figure 4-19 Unit Two Cable Tray Fire, 28,899 Mwh ...............................................................4-49
Figure 4-20 Unit Two Cable Tray Fire March 1999, 28,899 Mwh............................................4-50
Figure 4-21 Unit Four Air Preheater Guide Bearing Fire, 6,099 Mwh .....................................4-51
xv
LIST OF TABLES
Table 1-1 Component Description and Engineering Identification Number.............................1-18
Table 1-2 Phase 2 Recommendations and Actions Taken .....................................................1-36
Table 3-1 Boiler tube Failures, Causes and Recommended Action........................................3-15
Table 3-2 Typical Header and Steam Drum Problem Areas ...................................................3-18
Table 4-1 Cholla - Megawatt-hour loss summary report-fourth quarter 1999..........................4-28
xvii
1
BOILER RELIABILITY OPTIMIZATION PROJECT
PHASE 1, 2 & 4 AT AMERICAN ELECTRIC POWERS
BIG SANDY PLANT UNIT 2
1.1
EXECUTIVE SUMMARY
Preparing for competition American Electric Power (AEP) requested EPRI to undertake a Boiler
Reliability Optimisation project at their Big Sandy Plant. An availability of 95 % during peak
periods May through August and 90 % for the remainder of the year was set as a corporate
target. Meeting this target would give Senior Management the assurance that they could
compete successfully in the power generation market.
The project focussed on the long-term integrity of the water and steam touched tubing within the
boiler envelope, training on EPRIs tube failure reduction program and the provision of a EPRIs
Boiler Maintenance Workstation (BMW) software package for trending and tracking tube failure
data and information. The project scope also included training on EPRIs Streamlined
Reliability Centered Maintenance software using the fuel system as an example. An outage task
prioritisation model Risk Evaluation and Prioritisation (REAP) was used to prioritise the tasks
identified for the Unit 2 scheduled outage in April 2000.
This project, Phase II of EPRIs Boiler Reliability Optimisation project, commenced in March
2000 and was finalised with the installation of the BMW software and user training in July 2000.
Phase I was completed with the issuing of the final report in January 1999. Between March and
July 2000 a number of visits to the plant were made to interview and collect data for analysis.
This report focuses on the development of a failure defense plan for the water and steam
touched boiler tubing and contains a number of recommendations.
The main recommendation are listed below:
1. In the Technical and Organisational Assessment section there are a number of specific
recommendation. Collectively these are related to improving planning and scheduling of
boiler maintenance work. Thus, a high level recommendation would be to align the boiler
maintenance strategy i.e. frequency and duration of scheduled outages and inspection scopes
of work to the overall station performance goals of 95% availability during peak periods.
Specific boiler availability targets should be developed relating to the number of tube leaks
per year. One tube leak per unit per year, no repeat failures, no failures because of
sootblower erosion are considered reasonable targets. The strategy and performance targets
1-1

Boiler Reliability Optimization Project Phase 1, 2 & 4 at American Electric Powers Big Sandy Plant Unit 2
should be compiled into separate plant directives, which are reviewed annually for
effectiveness and appropriateness.
2. To repair and calibrate all boiler instrumentation critical to the efficient and effective
operation of the boiler. This includes air, gas, water and steam temperatures and pressures
and reheater and superheater metal temperatures.
3. The installation of on-line water and steam chemistry instrumentation and the establishment
chemical performance index. This index can the be used in a proactive way to ensure
compliance to EPRI water and steam chemistry guidelines.
4. To prepare detailed inspection plans based on the tasks identified in the Boiler Failure
Defence Plans presented in this report. These inspection plans should be loaded into the
CMMS. This would facilitate the development of a scheduled and or forced outage plan at
short notice.
5. Once the installation and the user training on the Boiler Maintenance Workstation (BMW)
have been completed, the BMW database needs to be updated with all the history of previous
tube failures. This will enable tube leaks and repairs to be tracked and trends to be
established. This will also assist in formulating focussed boiler inspections plans.
The plant has knowledgeable and skilful people. Given time, management support and coaching
to implement the recommendations highlighted in this report the plant team will achieve the
goals identified by Senior Management.
1.2
INTRODUCTION
During the period 1997 to 1999 AEP Big Sandy Plants availability loss was 7.9 % of which
4.2 % can be attributed to boiler tube failures. The causes of these failures varied however the
dominate repeat failures were due to fly ash erosion, oxygen pitting, long term over heating and
firing side corrosion. During 1999, in excess of $2.0 million dollars of lost revenue can be
attributed to outages on Unit 2. Concerned about this loss and a possible future increase in the
Equivalent Forced Outage Rate (EFOR) lead senior management at Big Sandy Plant to request
assistance from EPRI.
Participation in EPRIs Boiler Reliability Optimization program was proposed. The program
consists of several customised Boiler Reliability Optimisation Projects. Each project consists of
four phases that involve EPRI technical staff, with participation from the plants operating,
maintenance and engineering support staff. Each phase builds on the strengths of the preceding
phase. These phases are:
Phase I - Technical and Organisational Assessments
Phase II - Development of a Boiler failure defence plan
Phase III - Implementation of a failure defence plan
Phase IV - Continuous Improvement and Process automation
1-2

AEP Big Sandy Plants management chose to limit their participation to elements of Phase I.
This was completed in January 1999. Satisfied with the results, AEP Big Sandy management
requested EPRI to continue with the program, i.e. complete Phase I Technical and
Organisational Assessment, and implement a tailored Phase II. Phase II was limited to only the
water- and steam- touched boiler tubing. This report discuss the findings of this limited scope
project.
1.3
SCOPE OF WORK
This consisted of the following activities:

1.1 Complete Phase I Technical and Organisational Assessment.
1.2 Develop a Failure Defence Plan for water and steam touched tubing.
1.3 Provide Streamlined Reliability Centered Maintenance (SRCM) software and training, and
analyse two systems.
1.4 Evaluate and prioritise all outage tasks on an upcoming outage, using EPRIs Risk
Evaluation and Prioritise Program.
1.5 Provide Boiler Tube Failure Reduction and Cycle Chemistry Improvement training.
1.6 Supply, install and provide training on EPRIs Boiler Maintenance Workstation (BMW)
software for tracking boiler tube failure information.
1.4
FINDINGS AND OBSERVATIONS
1.4.1 Phase I and II - Technical and Organisational Assessment

A maintenance process assessment was conducted as a part of the work performed in the Boiler
Reliability Optimization Project. The purpose is to understand the current processes,
technologies, organizational strengths and weaknesses, and plant people to design a manageable
change within the implementation plan. Data was collected and evaluated and benchmarked
against best practices identified and tailored by EPRI. The assessment team spent one week on
site interviewing a cross section of plant staff to determine how boiler maintenance work is
identified and performed. The team focused on four main topics namely Work Process, Work
Culture/Management, Technology and People. The data and information obtained was
compared to industry best practice as identified in the EPRI Best Practice, to identify
opportunities for improvement.
This section of the report presents the findings and recommendations to align AEP - Big Sandy
Plants work process with Best Practice i.e. to move boiler maintenance from a reactive mode to
one that is planned. The benefits of moving towards a more planned approach will be seen in
reduced O&M cost and improving boiler reliability.
1-3

With this information and available resources AEP - Big Sandy Plant can become the Best
Practice Plant within the AEP organisation for which it has been striving to become. Given
time, management commitment and coaching, the plant team has the vision and can successfully
lead this plant into full implementation of the boiler reliability optimization processes. From the
data and interviews, strengths and weaknesses were plotted on a spider chart and
recommendations formulated
For a condition-based, highly planned, maintenance optimization program to be effective, it must
integrate the Work Process, Management and Work Culture, Technologies, and the People into
the total operation of the pant. Each category is equally important to the success of the program,
as illustrated in Figure 1-1.
E
C
P
E
O
O
R
P
LE
P
S
S
T
N
TE
C
E
M
E
G
A
N
O
N
A
LO
G
Y
PDM
Figure 1-1
Maintenance Optimization Program
To derive maximum value, all four categories and their sub-categories must be integrated to form
one cohesive program. The key sub-categories are listed below:
Work Process
Work culture/Management
Work Identification
Setting Goals
Leadership
Work Planning
Organisation
Accountability
Work Execution
Communications
Work Close-out
Global Metrics
Technology
People/Skills
Work Management System
Training
Qualifications
Diagnostic Technologies
Utilization
Communication
Integration Tools/Techniques
1-4

Figure 1-2 is the benchmarked Spider chart utilizing the Power Industry as the source database.
The outer circle at 8.0 represents best practice elements found in industry. The inner circle at 5.8
is the industry norm. Big Sandy Plant was evaluated and found to be at or near the industry
norm. Big Sandy Plant has second quartile performance when compared to industry benchmarks.
Information Integration, Work Execution, Metrics, and Utilization elements are slightly below
industry norms.
It should be noted that overall, Big Sandy has relatively equal strengths throughout most
elements. Information Integration can be improved in accordance with some of the
recommendations coming from this project. Metrics are also contained in this project.
Utilization will improve as more effective planning and a scheduling processes are put into
place. Big Sandy staff given the direction has the strengths that are necessary to succeed.
P M B as is
Q ualific ation
Com m unic ation Intra
8.00
7.00
6.00
W ork Identific ation

W ork Control
5.00
Utiliz ation
W ork E x ec ution
4.00
3.00
Training
W ork M anagem ent S y s tem s
2.00
1.00
0.00
CB M Tec hnologies
B enc hm ark ing
S etting G oals
Inform ation Integration Tools
Com m unic ation Inter

M etric s
Leaders hip
Continuous Im provem ent

A c c ountability
O rganiz ation
Figure 1-2
Big Sandy benchmarked against Industry Best Practice
1.4.1.1 Work Process

For a maintenance program to function cost effectively, all equipment maintenance work should
proceed through four distinct steps Maintenance Strategy, Work Identification, Work Control
(Planning and Scheduling) and Work Execution (including feedback) as shown in Figure 1-3.
1-5

Work Flow Process

PM
Basis
CBM
Planning
Scheduling
Work
Activities
Maint.
Strategy
Work ID
Work
Control
Work
Execution
Feedback CBM
Figure 1-3
Work Process
Using the Work Process Model - Figure 1-3 the assessment team produced a detailed model that
highlights the areas of responsibilities. These are shown in Figure 1-4.
1-6

PM Basis
Change Control
PM Basis
RCA
Boiler Specialist
PM's
Planners
Information
Integration &
Decision
CBM Coordinator
Technology Owners
Boiler Specialist
CAP
Technology Owners, CBM
Coordinator, RCA Process Owner
CD's, PR's
Boiler Specialist
Work Mgmt System

Operators
Work Order
Generation
CM's
Improvement
Work
Identification
CBM Data Collection &

Analysis
Initiator
Planning
Planners
Boiler Specialist
Work Control
Planners
Scheduling
Work
Execution
Work Execution
Boiler Specialist
Boiler Specialist
Craft
Post Maintenance
Testing
Close Out
Boiler Specialist
Boiler Specialist, CBM
Coordinator
Boiler Specialist
Planners, Craft
Figure 1-4
Work Process Model
Maintenance Strategy
The starting point for developing a maintenance strategy is to first determine a maintenance
objective. An objective should be quantified in terms of availability and performance. These
requirements should be determined jointly between the Maintenance and Operating and or the
1-7

Production Departments. Maintenance Management is then in a position to determine the best

mix of maintenance techniques to used, resources requirements and the costs thereof.
A maintenance strategy is then developed from these agreed objectives. It defines the process for
identifying and achieving sound operational objectives, selecting the maintenance approaches,
monitoring performance and providing effective management control. The strategy therefore
consists of a management and technical strategies. The Management strategy defines how the
business management skills are used to integrate people, policies, equipment and practises to
identify improvement opportunities. The technical strategy states how the technical knowledge
and experience are used to identify and implement the best proactive maintenance, repair, service
and replacement of all equipment in line with AEPs business performance objectives.
The maintenance strategy is built primarily from the preventive maintenance (PM) basis that
exists at the plant. The aim is to develop an optimized set of PM tasks that will ensure Big
Sandy Plant - Unit #2 Boiler will achieve its reliability targets.
Findings:
The Boiler Specialist owns the Boiler System PM Basis.
The Boiler System PM contains most of the inspection requirements, physical improvement
PMs, etc for work to be performed at various intervals on the various boiler components.
Boiler Specialist ensures that the technology owners complete their inspection PMs in
accordance with the PM Basis for the Boiler.
The Boiler Specialist actively participates in the Root Cause Analysis performed on the
boiler events.
Operating procedures have not been reviewed for some time and are out of date with current
practice.
Recommendations:
1. The PM Basis must be reviewed and updated to take into account the failure mechanisms
identified during the past outage inspection.
2. The Boiler Specialist should have an Operating Department counterpart who would be
responsible to review and update and align operating procedures to current practice.
3. Boiler tube failures/Root Cause Analysis should be tracked and trended.
4. The boiler inspection plans need to be updated to take cognise of the change in the frequency
of boiler periodic outages.
Work Identification
Work on equipment is identified from one of the following three sources of information:
1-8

Corrective Maintenance Repair tasks (CM) - tasks as a result of loss-of-performance, component

breakdown, or catastrophic equipment failure that must be attended to immediately. These tasks
are generally not scheduled.
Preventive Maintenance Tasks (PM) - tasks that are time based recurring work that is deemed
necessary to keep equipment in optimum running condition. Implementation of this work should
be both planned and scheduled.
Predictive Maintenance tasks (CD) - tasks that are condition directed based on information
derived from a variety of condition information sources. These sources are identified and
controlled on an E&CI matrix. All condition directed efforts to maintain equipment should be
both planned and scheduled.
Findings:
Prioritization seems to be fix it now or fix it later. When dealing with boiler issues,
which are mainly outage tasks, value of work concepts should replace prioritization.
The work process is split into two separate planning windows. One for emergency work i.e.
works requiring immediate attention and the other for normal work, which passes through a
daily scheduling activity.
Emergency maintenance work is discussed between operations and maintenance leaders,

while all other work orders (CM, PM and CD) are reviewed, selected and assigned to be
executed by the PSLs.
The various Crafts persons walk the job down and plan job requirements.
There is no formal root cause analysis process in place to ensure failures are thoroughly
investigated to prevent recurrence.
Recommendations
1. A formal RCA program will result in additional information valuable in managing the Boiler.
2. The roles and responsibilities of the PdM Co-ordinator should be clearly defined and should
include the following aspects:
Ensure data collection routes and analyses are performed on schedule.
Conduct pre-outage inspections.
Post-maintenance testing.
Working with system owners to integrate condition based data/information
Develop and maintain a condition status report.
Document case histories and cost-benefit calculations.
Ensure PDM results are used to moderate the PM program extend periodicity, PM
activity modifications, removing redundant PM etc. Thus taking full advantage of the
CBM program.
1-9

3. Infrared thermography should be used on the boiler to identify casing leaks, valve passing
etc.
4. Develop an Equipment and Condition Indicator matrix (E&CI) for all critical boiler
equipment/components such as in Figure 1-5. This will assist the equipment owner to
assessing the condition of equipment under his control.
Work Control
The work control function shown in Figure 1-3 is an important function when focussing on
reducing O&M costs and simultaneously increasing reliability
One of the activities when optimizing work control is to review and purge if necessary the
corrective maintenance (CM) component of the workload. CM negatively impacts the ability to
meet performance goals of reducing O&M cost performance and Equivalent Forced Outage Rate
(EFOR). Decreasing the amount of CM does not mean that equipment failures will no longer
occur. Rather, it means that management will be actively involved in deciding whether the best
economic decision for the station is to allow a specific component, with known problem
conditions to run-to-failure; or, should plans be established to effect the repair at a convenient
time.
Findings
The frequency of major outages has been changed to every 3 years with an inspection/minor
repair outage of short duration at approximately 18 months.
Outage work scopes are agreed to prior to the outage. However, a considerable amount of
outage work is uncovered during the outage with the inspections. Review of the data
available prior to the outage indicates some of the inspection results could have been
predicted and repairs planned.
Planners prepare work packages for critical work orders.
Boiler Specialist serves the role of QA/QC for all boiler activities whether performed by AEP
forces or contractors.
Forced outages are planned, reviewed, and published once per month. Not all items on the
forced outage list are fully planned and trigger ready.
Forced outage plans are coded in the PIMS and maintained by planners.
Recommendations
1. In order for the 3 yearly boiler outages to be successful a detailed inspection plan needs to be
developed by the boiler specialist to all known and potential problem areas are adequately
inspected and repaired to ensure a high availability between periodic outages.
2. Develop a four-week rolling scheduling process for all maintenance to assure high utilization
of resources.
1-10

3. Include post maintenance testing requirements into the planning process. This is a must for
most CD work orders.
Work Execution
This is the final activity and includes work close-out.

Findings
AEP support personnel and contractors do a reasonable job judged by the small number of
QA/QC issues raised and re-work done.
Pad welding and temporary repairs are tracked and noted to enable final repairs to be done
during a suitable planned outage.
Limited post maintenance testing is done.
Work close out is performed by the PSLs and at times without the knowledge of the Boiler
Specialist.
Recommendations
1. System owners and the Boiler Specialist should review close-out information for
effectiveness.
2. Compile a management directive identifying the minimum acceptable standards to be

complied with when performing boiler maintenance or repair work. This would include such
statements as every tube leak/failure should be investigated, pad welding should be
discouraged etc.
1.4.1.2 Technologies
An Optimisation program is made up of people, management systems, component owners,
technology specialists and predictive maintenance tools or equipment. The key predictive
maintenance tools used in a Boiler Reliability Optimisation project are infrared thermography,
vibration, oil analysis, ultra-Sonics and plant process data and information such as air, gas water
and steam temperatures and pressures, metal temperatures and water and steam chemistry data.
Equipment owners and technology specialists collect and collated data on various boiler
components. This data is organised and presented in an Equipment and Condition Indicator
Matrix. The matrix specifies the predictive maintenance technology to be used, the equipment to
be monitored and when the data should be collected.
The how, what, and when of condition data analysis is based on exceeding preestablished threshold levels and this information is presented in an Event Report and in an
Asset Condition Status Report. Analysis of the data and diagnosis of the cause incorporates
all available information including diagnostic data from all related technologies, operating
conditions from process data, operating log information, maintenance history, design information
1-11

and knowledge of the experienced personnel. Condition Assessment Reports should list, by
asset, any marginal or critical determinations by each technology, and include recommendations
for action based on the integrated analysis of the data.
The what, how and when of action is based on analysis of the data, and this establishes
what action is to be taken to correct the anomaly, how the work should be accomplished, and
when the work needs to be done to prevent an unplanned failure.
Determining the threshold levels to declare an event is based on technology standards and
deviations from baselines for the monitored equipment. Therefore, it is necessary to establish
baselines for all equipment in the PDM Program, and re-baseline the equipment following any
related repairs by performing post-repair acceptance testing.
Findings
Draught gauges and metal temperature thermocouples were found to be defective.
No on line sodium analysers for continue monitoring of steam and water quality was
available.
Recommendation
1. Repair and calibrate all essential/critical boiler instrumentation.

Work Management System
Findings
PIMS meets the basic needs of the organization and has the capability to capture boiler
inspection data.
Recommendations
1. PIMS should be expanded to capture all boiler inspection data

Maintenance & Diagnostic Technologies
Findings
NDE Inspections are consistent with industry practice
Limited boiler performance testing and trending of results is done.
Recommendations
1. Establish technology owners with responsibilities to include condition analysis.

1-12

2. Need to have a more comprehensive PDM reports as health report not exception reports.
Information Integration
The key to utilizing the condition-based information in an effective manner is the ability to
quickly access the information and to trend the data over time to enable a prediction of when
work should be performed. EPRIs Boiler Maintenance Workstation (BMW) is one such system,
which can be used to track and trend boiler tube failure data and information. Another useful tool
is the WEB based Condition Status Report software. The software has the capability to store
equipment data and information, capture recommended actions, and report on the overall
condition of the critical plant equipment in detail. From this information, management can
determine whether the overall condition of the plant is deteriorating or improving by tracking the
history and costs of the equipment. Decisions made and or action taken and work order numbers
are also recorded. Post maintenance test results are recorded to determine whether the diagnosis
and maintenance was appropriate.
Findings
There is no integrated condition status reports produced on the boiler or in fact any other part
of the plant.
Recommendations
1. Formalise monthly Boiler Condition Based Maintenance (CBM) meetings. Attendance by

Equipment/Component and Technology owners, PdM Co-ordinator, Operating and
Maintenance staff and the Boiler Specialist to make maintenance decisions based on the
condition of the equipment/component. Once a year, perhaps prior to any budgeting cycle,
the Boiler Specialist should present a Boiler Failure Defence Plan indicating the current
condition, short and long term activities/tasks that need to be undertaken to ensure boiler
reliability.
2. Case histories and cost benefits analysis of using PDM technologies should be document
either in the BMW or PIMS databases.
1.4.1.3 Management and Work Culture
The objective is to develop a work environment in which an effective combination of people,
work processes and technologies are used to achieve the overall business goals. To achieve this
management must exhibit leadership i.e. the ability to delegate responsibility and authority, and
to hold people accountable for their performance. Leadership also includes defining
organisational direction, setting realistic measures and targets and communicating these through
out the organisation. Measures (metric) can be lagging measure the output of the process or
leading a measure of the effectiveness of the elements of an organisation.
1-13

Recommendations
1. Roles and responsibilities must be clearly defined and understood by the individuals in the
plant.
2. Measures (metrics) and targets need to be developed for the boiler. These metrics could be
the following:
# of tube leaks per year suggest one tube per boiler per year,
no tube leaks caused by sootblower erosion,
no repeat boiler tube leak failures,
boiler availability,
boiler forced outage rate as a contribution to the overall forced outage rate and
long term plant health measures like a thermal index which is a measure of the
equivalent number of operating hours or extra life consumption experienced by boiler
headers as a result of operation at metal temperatures in excess of design.
These measures and targets should be specific, measurable, achievable within a specified
period of time and reasonable and should be communicated through out the organisation.
3. An annual review of the maintenance program should be undertaken to determine its
effectiveness i.e. the right technologies are being applied to the right assets at the right
frequency to meet the overall business objectives and maintenance strategy. These
assessments should validate the following questions:
Were there unexpected or premature failures that were not detected by at least one
technology? This could result in applying PDM to an asset that was not previously
being monitored, or increasing the frequency of monitoring on a previously
established route, or indicate the need for a new PDM technology, or readjustment of
a CM Task.
Were failures occurring that were detected but without sufficient time to take action?
This could result in adjustment of alert threshold levels.
Were there long periods of operation without any developing problems that contained
numerous condition data sets? This could result in decreasing the frequency of
monitoring.
Was a technology ineffective in detecting an emerging problem? This could result in
removal of an ineffective technology from the program.
Did the costs, measured in dollars, exceed the benefits of the program? This could
signal the need to redesign the program.
Was all of the data collected analyzed?
Was action taken on detected events? This could result in a refocus of the program
leaders and a re-aligning of sponsors.
Did the program fail to meet the established goals? This could result in a redesign of
the program.
1-14

4. Consideration should be given to having an annual Boiler Specialist Meeting and forming a
Boiler Specialists Users Group. This will aid in the transferring of information and
knowledge amongst the Boiler Specialists within the AEP organisation.
1.4.1.4 People
Human beings are an important asset. Their reliability making and keeping commitments
has a huge impact on plant reliability. All equipment can in some way be traced back to human
beings. If human beings are reliable, your equipment will be reliable.
Findings:
Training meets the basic needs of the plant. New equipment training for the craft is lacking.
Communications among PSLs is good.
Good team work.
Utilization is lower than expected. This is as a result of inadequate planning and staff has to
wait for other disciplines.
Recommendations
1. Consider providing Level of Awareness training (LOA) training for the entire staff on Plant
Maintenance Optimization.
2. Re-train all PIMS users on the details of PIMS and how best to it.
1.4.2 Failure Defence Plan Water and Steam Touched Tubing
To have an effective defence plan, it is important to uniquely identify Equipment Identification
Number (EID) all the various sections and/or components to be included in the Defence Plan.
This number is then used in the Computerised Maintenance Management System (CMMS) to
track inspection details and trends. Table 1-1 shows all the various tubing sections of the boiler.
Table 1-2 represents the Failure Defence Plans for the various sections of tubing listed in Table
1-1. These plans were developed from a detailed boiler inspection and discussions with the
Boiler Specialist. They identify the failure mechanisms, direct causes, short and long-term
actions to be taken to prevent or at least minimise the failure and or damage. These action plans
also give the various NDE and NDT techniques to be used to identify and quantify damage by
the various failure mechanisms and the criteria to be used when using performing boiler
maintenance and or repair work.
The following failure mechanisms were identified as being active in the various tubing sections:
Sootblower erosion
Fly ash erosion

1-15

Fire side corrosion
Corrosion fatigue
Dissimilar metal welds
Oxygen pitting
Short term overheating
Long term overheating
Findings
Economiser
No significant tube failures have been experienced in the economizer.
The economizer is of the ringed tube type and suffers from pluggage especially on the North
side. Baffles have been installed to equalize gas flow and minimize ash pluggage.
Clean air velocity tests have not been done to verify the correct position of the baffles.
Plans are in place to open up gas passes to minimize pluggage.
One leak was encountered last year due to flyash erosion. Tube shields were installed 2 years
ago to minimize flyash erosion.
No header cracking has been noted.
Waterwalls
The ash hopper slope has channelling between the membrane and the tube in the tube
material. About 82 tubes have been found with some amount of material loss due to slag
erosion. No leaks have been attributed to this problem. Slagging of these tubes has not been
a problem since changing to low NOx burners in 1994.
The sidewalls have experienced severe wall loss since conversion to low NOx burners.
Portions of the sidewalls were overlaid in the fall of 1999 with Type 312 SS to mitigate this
problem. No further testing has been performed since this inspection. The unit has only
experienced one trip since the outage and did not stay off line long enough to allow for
testing. This overlay will be inspected in March and an additional 500 sq. ft. will be
overlaid.
Since the overlay outage, testing has been performed using 3/8 tubing inserted through the
wall membrane during the outage on the left side. Initial measurements showed no oxygen
during operation. Burner modifications and adjustment have raised the oxygen content to a
positive level. Plans are to install several more taps into the firebox during the next outage to
help facilitate future burner adjustments. Currently measurements are performed manually
on the sample positions. Automatic sampling should be developed and tied into the database
with feedback to controls or maintenance.
The upper furnace has experienced no problems.
1-16

The center division wall has experienced no problems.
One failure has been reported in the roof due to erosion from a casing leak. The attic has
since been pressurized and no more leaks have been observed.
Several areas of casing/membrane leaks were noted in the upper firebox. These should be
repaired at the next outage to prevent erosion from ash or casing damage from heat.
Screen tubes have had some erosion from flyash and sootblowers. Tube shields have been
tried with no success. The screen tubes are carbon steel with a wall thickness of 0.300.
Some pad repair or weld overlay work is expected next outage. More information is needed
on this problem including UT thickness readings. This wear is probably from water in the
sootblowing system but could be aggravated by large coal size, late combustion and LOI.
Plant staff indicated coal mill fineness tests were performed twice a year.
No severe problems were noted with the waterwalls other than the wastage due to low NOx
burners. This is being addressed with the weld overlay and burner adjustments.
Platen Superheat
Approximately 300 DMW are located in the platen superheat section and are scheduled for
replacement using EPRI developed technology.
Finishing Superheat
Some sootblower erosion has occurred in this section. Some tubing has been pad welded to
restore minimum wall. No leaks have occurred due to erosion. Alignment handcuffs in this
area are in need of repair. Current plans are to install heavier handcuffs next outage to keep
tubing in alignment and avoid flyash erosion.
No plugging problems have been noted in this area except when sootblowers are out of
service. The tubing in this area was too long when installed and hit the slope when hot. This
tubing has been shortened. No operational problems are noted at this time.
First Reheat (High Pressure)
This area probably contains the most damage of any part of the boiler. Past lay-up
procedures allowing moisture to condense in this area has led to oxygen pitting in the
horizontal sections.
Current lay-up procedures of using nitrogen blanketing, drying out with dried air, or heating
up to dry after coming off line are believed to have stopped the damage.
The second or third tube in the bundle has suffered from long term overheat fish-mouth
type failures. Other AEP 800 MW plants have replaced this tubing with P91 material. A life
assessment using EPRI TUBELIFE and other NDE should be performed on this tubing. If
tubing requires replacement, T91 or other new upgraded material such as T23 should be
used.
Damage is also occurring in this section from vibration fretting of the cane tube spacers.
Egg shaped spacers are placed vertically between tubing bundles and turned until tight. Then
1-17

straps are welded between adjacent spacers to hold in place. No attachment is made to the
boiler tubing. During operation these spacers become loose and allow the tubing to vibrate,
fretting the tubing and causing leaks. AEP is currently not replacing spacers when they fall
out.
Second reheat (low pressure)
The outlet header and top two sections of tubing have been replaced with SA 213 T-91
material in the highest temperature areas. No other leaks or failures have occurred in this
area.
Spray Attemperators
The nozzles have been inspected and replaced. The liner has been replaced on the Main
Steam Attemperator. This equipment is currently on a seven-year inspection cycle. This
inspection frequency should be monitored and adjusted as required. Liners should be
inspected with boroscope. Nozzles should be removed and inspected. Water stop valves
should be inspected to prevent thermal shocks to the superheat and reheat piping.
Recommendations
1. Develop a series of inspection tasks using the data and information identified in the Failure
Defence Plans in Table 1-2. These Defence Plans need to be reviewed and converted into
specific action tasks with allocated responsibilities for completion. The inspection type tasks
need to be formatted and standardised and put into the Computerise Maintenance
Management System database for reference and future boiler inspections.
Table 1-1
Component Description and Engineering Identification Number
#
1-18
EID
DESCRIPTION
5512230200 ECON 1
Economiser Tubes U2 1 Reheater side
5512230200 ECON 2
Economiser Tubes U2 2 Reheater side
5512240200 DIVWL
Waterwall Tubes U2 Division wall
5512240200 LFFWL
Waterwall Tubes U2 Lower furnace front wall
5512240200 LFSWL
Waterwall Tubes U2 Lower furnace side walls

(incl. 40 tubes front and rear)
5512240200 LFRWL
Waterwall Tubes U2 Lower furnace rear wall
5512240200 UFRWL
Waterwall Tubes U2 Upper furnace rear wall
5512240200 UFFWL
Waterwall Tubes U2 Upper furnace front wall
5512240200 UFSWL
Waterwall Tubes U2 Upper furnace side walls
st
nd

10
5512243200
Screen Tubes U2 Furnace
11
5512241200 HRAT
Heat Recovery Area (HRA) U2 Incl. Side, front

and rear walls, partition wall and roof
12
5512241200
Aperture Tubes U2 Incl. Side walls, floor and

screen
13
5512240200 FROOF
Waterwall Tubes U2 Furnace roof tubes
14
5512210200
Platen Superheater
15
5512212200
Finishing Superheater
16
5512220200
1 Reheater
17
5512222200
2 Reheater
st
nd
1-19
Components 1 through 16 (all components within sootblower range)
Malfunction of a sootblower or 1
the sootblowing system
because of incorrect operation
and/or inadequate
maintenance (control logic sequence, operating
temperature and pressure,
mechanical misalignment,
drainage pipe slopes etc)
1-20
Preventive Measures
Short/Long Term
NDE Detection Technique
Sootblower Erosion
L M N O P Q #
Fire Side Corrosion
Falling Slag Erosion
Caustic Gouging
Coal Particle Erosion
Acid Phosphate Corrosion
Short Term Overheating
Supercritical Waterwall Cracking
Low Temperature Creep
Corrosion Fatigue
Fatigue
Acid Dewpoint Corrosion
Pitting
A B C D E F G H
Hydrogen Damage
Fly Ash Erosion
Possible Causes
Thermal Fatigue
Corrosion Corrosion
CRITERIA
Confirm mechanism by
VT
inspecting tubes around and in /EMAT
the immediate vicinity of the
blower.
Check for tube wastage - flat and/or

'shiny' surfaces or if the boiler has been
washed check for a fresh layer of rust and
gouges from steam cutting.
Determine the extent of the UT

erosion by mapping the tube
thickness in the affected ares.
Compare with previous
readings and calculate an
erosion rate and trend results if applicable.
Use AEP's Standard for repairing or

replacing tubes. Typically 70% of wall
thickness is used. Do not pad weld to
restore thickness for < 70% because of
the potential to introduce other
detrimental failure mechanisms - copper
embrittlement and hydrogen damage. If
root cause of erosion has been
determined and adequately addressed no
additional erosion should place.
Check, correct and verify the VT

operation of moisture traps
and remove remaining orifices.
Check, correct and verify
VT
blower or system operation to
ensure correct sequence,
frequency, angel of rotation,
travel, misalignment, operating
steam temperature and
pressure.
Check, correct and verify
VT
steam supply (pressure and
temperature), drainage slopes
and piping insulation.
Use the OEM's manual or site specific

procedure.
Sootblower Erosion
L M N O P Q #
Fire Side Corrosion
Caustic Gouging
Corrosion Fatigue
Fatigue
Pitting
A B C D E F G H
Hydrogen Damage
Fly Ash Erosion
Possible Causes
Thermal Fatigue
Corrosion Corrosion
Preventive Measures
Short/Long Term
CRITERIA
Emergency repairs such as

pad welding should be
replaced at the next
opportunity.
Develop a map identifying the
exact location of each pad
weld to ensure all locations are
recorded.
Optimize blowing sequence
to ensure an effective boiler
sootblowing program.
VT
Establish and implement a

routine maintenance schedule
to inspect and test blower
operation.
1-21
Component # 1, 2 and 12 through 17
Primary causes are excessive 2

local velocities and/or dust
burden.
Excessive
local velocity
design/configuration
distorted/misalignment tubes
- Gas flows above design
rating
Dust burden
- Increase in erosive particles
ie increase % ash
- Sootblower operation
Other
- Incorrectly install shields
and/or baffles
Inappropriate and/or incorrectly
applied coatings
- Broken or misaligned "canes"
and "handcuffs.
1-22
Preventive Measures
Short/Long Term
Sootblower Erosion
L M N O P Q #
Fire Side Corrosion
Caustic Gouging
Corrosion Fatigue
Fatigue
Pitting
A B C D E F G H
Hydrogen Damage
Fly Ash Erosion
Possible Causes
Thermal Fatigue
Corrosion Corrosion
Confirm mechanism and

UT VT
determine the extent by
completing a dirty and a clean
inspection. The dirty
inspection will highlight areas
where excessive local
velocities occur. These can be
comfirmed by a detailed clean
inspection. The areas mostly
likely to show signs are at the
base of the fins of the first
economiser bank, the element
ends between tube and wall
and adjacent to any flow
staighteners and/or baffles.
Perform a CAVT to confirm
high velocity areas. If high
velocities are detected - design
and install baffles or flow
straighteners. Perform a
second CAVT to confirm actual
velocity profile.
If baffles and/or flow
straighteners have been
installed perform an annual
inspection to determine the
effectiveness of the
modification and adjust if
necessary.
CRITERIA
Use AEP's standard for tube

replacement (<70% wall for waterwall
tubing and <85% for steam tubing) in
conjunction with EPRI's remaining life
assessment calculations results from
Level I or II as judgement criteria - TR111559. If trend data available, replace
tubing that will drop below wall thickness
criteria before next outage. If erosion
correction action taken and confirmed to
eliminate future wall loss, can leave
tubing in service with >70 % but <100%.
Preventive Measures
Short/Long Term
Repair, replace and/or align

damaged tubes and/or
elements.
Sootblower Erosion
L M N O P Q #
Fire Side Corrosion
Caustic Gouging
Corrosion Fatigue
Fatigue
Pitting
A B C D E F G H
Hydrogen Damage
Fly Ash Erosion
Possible Causes
Thermal Fatigue
Corrosion Corrosion
CRITERIA

replacing Tubes
For major damage, plan for

element/bank replacement.
If damage is at the base of
the fins on the first bank and
replacement is necessary then
consider designing out the
problem by installing straight
tubes.
See EPRI document for economizers

GS-5949
Repair, replace and/or align

damaged "canes" and
"handcuffs".
Check for and remove any
ash build up/plugage in the
economizer tube banks.
70 % through 200 mesh
1-23
Components 3, 4, 5 and 6
Primary cause is a
substoichiometric (reducing)
enviroment as a result of poor
combustion - burner air and
fuel distribution and incorrectly
set burner registers.
Secondary causes are related
to fuel composition in particlar
the amount of chlorine and
aggressive ash and also flame
impingement.
1-24
#
3
A B C D E F G H
Preventive Measures
Short/Long Term
Sootblower Erosion
Fire Side Corrosion
Caustic Gouging
Possible Causes
Hydrogen Damage
Corrosion Fatigue
Fatigue
Pitting
Fly Ash Erosion
Thermal Fatigue
Corrosion Corrosion
CRITERIA
L M N O P Q #

UT Use AEP's standard for replacement ie
extent of wall thinning by
/EMAT < 70% wall.
inspecting the area around the
CO level of < 1% and/or Oxygen levels
burners (approx 2 diameters)
of < 0.1%
and on the side walls at
Wastage rate of 12 mills/yr is considered
approx. midway between front
excessive.
and rear walls. these surveys
should be conducted in
Use AEP's standard for repairing or
conjunction with remain ing life
replacing tubes
assessment techniques to
determine wastage rates. Also
monitor/measure the levels of
Oxygen and CO near the
tubes on the side walls.
For damage, repair by the
use of high chromium weld
overlay material in affected
areas to resist corrosion
attack. Optimize combustion
to minimize the affect of
substoichiometric conditions
and institute long-term
monitoring - inspections and
oxygen and CO at or near the
tube walls.
Optimize combustion to
minimize the affect of
substoichiometric conditions
and institute long-term
monitoring of oxygen and CO
at or near the tube walls.
A B C D E F G H
Preventive Measures
Short/Long Term
Sootblower Erosion
Fire Side Corrosion
Caustic Gouging
Possible Causes
Hydrogen Damage
Corrosion Fatigue
Fatigue
Pitting
Fly Ash Erosion
Thermal Fatigue
Corrosion Corrosion
CRITERIA
L M N O P Q #
Inspect, repair or replace
damaged burner quarls.
Components 4 and 6 (hopper slope tubes and areas of high

heat flux)
Develop a spreadsheet to be
use to collect and trend
thickness measurement data.
Cyclic stresses brought about 4

by improper start-up and shut
down practices and
Poor water chemistry control
especially the control of ph
during shut down and start-up.

VT UT Using specialised UT/EMAT test for ID
initiated cracks. If cracks are > 50%
inspection and sampling.
through wall then replace tubing.
Remove sample attachment
tubes and areas of high stress.
Replace damaged tubes.
Modify tube attachments to
relieve obvious stress raisers.

replacing Tubes
Ensure compliance to AEP's

water chemisrty practices.
Ensure compliance to startup and shut down procedures.
Ensure compliance to AEP's
chemical cleaning procedures.
Correct tube spacing
Set inspection levels to
determine the effectiveness of
any modification and monitor
damage accumulation.
1-25
Preventive Measures
Short/Long Term
Component # 14
Possible Causes
Higher than expected metal
temperatures (brought on by
overfiring conditions) and stresses
- both thermal and differential
expansion
1-26
#
5
A B C D E
F G H
K L M N O
P Q
Fly Ash Erosion
Long Term Overheat
Graphitization
Chemical cleaning Damage
Pitting
Fatigue
Rubbing/Fretting
Sootblower Erosion
Stress Corrosion Cracking
Dissimilar Metal
Short Term Overheat
Creep
Fireside Corrosion
CRITERIA
#
VT UT
determine the extent. Map
location and inspect all DMWs.
Replace DMW with a
"dutchman"of upgraded material.
Depending on the specific
circumstamces re-positioning the
DMW may be more cost
effective.
Perform a remaining life
assessment calculation (Level I, II
or III).
Inspect and correct tube
supports, tube bindings etc
leading to axial stresses at welds.
Ensure functionality of metal
and steam temperature
thermocouples.
Use AEP's stanard for tube insert- 18 inches

preferably 36 inches.
Use EPRI's standard for weld geometry.
DMW life code from SIA (based on EPRI

PODIS code)
Preventive Measures
Short/Long Term
Component # 17
Possible Causes
Oxygen-saturated stagnant
condensate as a result of a unit
shutdown - forced cooling and/or
imprper draining and venting and
aggrevated by the
Carryover of sodium sulfate
from the reheat steam (Sodium
Sulfate tends to form at typical
reheat pressures)
#
6
A B C D E
F G H
K L M N O
P Q
Fly Ash Erosion
Long Term Overheat
Graphitization
Pitting
Fatigue
Rubbing/Fretting
Sootblower Erosion
Dissimilar Metal
Short Term Overheat
Creep
Fireside Corrosion
CRITERIA
#
VT UT
determine the extent by inspection
and sampling in particular areas
where the tubing has sagged allowing condensate to
accumulate.
If root cause has been identified
and rectified, minor pitting
damage can be left alone ie there
is no need replace tubing.
Continue to monitor and take
samples for analysis to check for
deterioration.
For major damage, plan to
replace, modify operating
procedure and institute long-term
monitoring.
Review and correct if necessary
the effectiveness of the shut down
and lay up procedures.
Use the remaining life assessment

calculations results from Level I or II as
judgement criteria.

replacing Tubes
Enforce compliance to shut

down and lay up - nitrogen
blanketing procedures.
If the lay up procedure nitrogen blanketing cannot be
used because of boiler work,
some other means of drying must
be used to ensure a moisture free
enviroment in the reheater.
1-27

Preventive Measures
Short/Long Term
Component # 3 to 17 (water and steam circuits)
Possible Causes
Exfoiliation induced: Caused by
the natural process of growth and
exfoiliation of oxides. These
oxide flakes tend to collect at
sharp bends.
Mainenance induced: Caused by
poor quality control during
welding, improper/incorrect
chemical cleaning or poor
combustion resulting in flame
impingment on water walls.
Operatging induced: Caused by
improper start-up and shut down
practice (not boiling out
condensate collected in the
loops/bends) and overfiring to
compensate for low feedwater
temperature.
#
7
A B C D E
F G H
K L M N O
P Q
CRITERIA
#
Confirm cause by either an
VT MT
analysis of a sample of the debris
or start-up and/or shut down
records. Confirm extent through
inspection of tube in the
immediate vicinity. Check for
swelling and cracks.
Check to ensure blockage has
been removed by using
compressed air or water.
Repair by inserting a 36 inch
Use AEP's Standard for chemical cleaning
"dutchman".
Ensure compliance to start-up
and shut down practices.
Ensure compliance to start-up
and shut down practices.
Check combustion and or
burners are optimized to ensure
no flame impingment is taking
place.
1-28
Fly Ash Erosion
Long Term Overheat
Graphitization
Pitting
Fatigue
Rubbing/Fretting
Sootblower Erosion
Dissimilar Metal
Short Term Overheat
Creep
Fireside Corrosion
Check drainage slopes and tube

bundle sagging. Correct if
possible, if not possible then to
check for any deterioration by
inspection and sampling.

replacing tubes
Preventive Measures
Short/Long Term
Component # 17
Possible Causes
Prolonged operation at high
temperatures (design operating
temperature and above)
#
8
A B C D E
F G H
K L M N O
P Q
Fly Ash Erosion
Long Term Overheat
Graphitization
Pitting
Fatigue
Rubbing/Fretting
Sootblower Erosion
Dissimilar Metal
Short Term Overheat
Creep
Fireside Corrosion
CRITERIA
#
VT UT
inspection. Perform UT metal
and ID oxide thickness
measurements.
Remove several samples both
across and within the bank for
analysis and replace withn a 36
inch "Dutchman"
Perform a remaining life
assessment calculation (Level I, II
or III).
For major damage, plan to

replace, modify operating
procedure and or design and
institute long-term monitoring.
Perform hydraulic test as an
integrity check.

replacing Tubes
Use the remaining life assessment

calculations results from Level I or II as
judgement criteria- EPRI's TUBELIFE
program.
Replacement should be T91 or equivalent
material such as T23.
Compliance to ASME code requirements fo r

test pressures and temperatures. Increasing
pressures above code to force failure is a poor
practice and fruitless as crack > 50% through
wall can tolerate design pressures without
failure.
Calibrate and ensure

functionality of all installed metal
temperature thermocouples.
Set inspection levels to
determine the effectiveness of any
modification and monitor damage
accumulation.
1-29

1.4.3 Streamlined Reliability-Centered Maintenance

Streamlined Reliability Centered Maintenance is a logical, systematic, functionally based
methodology used in the evaluation of a facilitys system or unit. The evaluation includes a stepby-step consideration of the integral functions related to the operation of the system or unit. The
failure mechanisms of each of these functions, the effects of the failure and the selection of
appropriate and effective maintenance tasks to mitigate these failures are identified. The goal is
to develop a cost-effective maintenance program, based on system functionality, which will
enhance system reliability. This makes optimum use of available maintenance resources and
provides a documented base for future additions/revisions to the maintenance program.
Two systems were reviewed steam- and water-touched tubing and the fuel system. Appendix 1
gives the results of this analysis.
These results still need to be compared with those in Computerised Maintenance Management
System. The purpose of this comparison is to identify changes to the existing program, to
optimise the Fuel system preventive maintenance program. The comparison also provides
another check of the analysis to ensure completeness and validity of assumptions.
Once the comparison is complete and has been approved the new tasks can be incorporated in
the CMMS.
1.4.4 Unit 2 Outage Task Prioritisation
An EPRI Risk Evaluation and Prioritisation (REAP) model was used to demonstrate how outage
tasks can be evaluated and prioritised. Thus providing a documented basis for decision making.
The model is based on determining the risk of performing or not performing a preventive or
corrective maintenance task. This is achieved by scoring a number of answers to questions about
the task at hand. Labour hours and cost are also used to determine the value and cost of the task.
The following is the results of a REAP analysis performed using the Unit 2 outage task list. A
total of 119 maintenance outage tasks were reviewed. These tasks had been determined from a
much longer list, which had been subjected to a gut feel prioritisation method. Unfortunately
the total list was not available to compare the gut feel method verse the approach taken in the
EPRI model.
Figure 1-5 represents the resultant scatter diagram of all 119 tasks analysed.
1-30

T a s k C o s t V a lue S c a t t e r
1.E+09
V alue/C o s t O p timiz atio n Line s
1.E+08
1.E+07
1.E+06
1.E+05
1.E+04
1.E+03
1.E+02
1.E+01
1.E+00
$100
$1,000
$10,000
C os t
Figure 1-5
Task Scatter Diagram - Value to Cost
The angled lines in figure 1-5 are generated by recognising that outage work scopes are built by
starting with the most valuable least cost work and continuing to add more expensive valuable
work until all to work has been included. Integrating all this work results in the Accumulative
Value and Cost curve Figure 1-6.
140
5.E + 09
4.E + 09
4.E + 09
3.E + 09
3.E + 09
2.E + 09
2.E + 09
1.E + 09
5.E + 08
0.E + 00
120
100
80
60
Task s
40
V alue
20
No. of Ta sks
V a lue
N o. of Tasks & V alue vs C ost
0
0
20000
40000
60000
80000
100000
Costs
Figure 1-6
Accumulated Value verse Cost
1-31

Figure 1-6 represents the accumulated value versus cost versus number of tasks and shows that
at the 32,700 cost level (cost is number of man-hours), 3.74 E09 value is captured with 95 of the
119 outage tasks. In other words with 40% of the outage task hours, 95% of the outage task
value can be captured with 80% of the tasks being performed. The vertical lines show where
there is a significant slope change. In this case they show perhaps where one can draw the
line on an outage work scope i.e. where 80 % of the value is captured with 20 % of the cost.
At that threshold, the following tasks would not be performed:
AIR HEATER SEALS
3356430
AIR HEATER SEALS
3423125
AIR HEATER SEALS
3381022
AIR HEATER SEALS
3423136
4KV ROOM WALL
3374302
INSPECT GAS OUTLET DUCTS
3422425
CIRC WATER PUMP DISCHARGE EXP. JOINTS
3356636
DEAERATOR INSPECTION
3422484
OVERLAY SIDEWALLS
3357443
NDE ON FACTORY WELDS 1ST RH PIPING
3374254
PA & FD FANS - CLEAN & INSPECT
3422941
BOILER DIVISION VALVE-OPEN, INSPECT & REPAIR
3420561
COAL CHUTE RENEW WEAR PLATES
3422915
COAL CHUTE RENEW WEAR PLATES
3422904
REPLACE COAL CHUTES
3415274
REPAIR PENTHOUSE CASING LEAKS
3421165
REPLACE 80 TUBES REAR ASH HOPPER SLOPE
0577695
REPAIR APERATURE MEMBRANE
3365526
801 ASH GATE
3358014
BURNER REGISTERS - INSPECT & REPAIR
3420966
BURNER REGISTERS - INSPECT & REPAIR
3420970
GENERATOR INSPECTION
3374350
MAIN TURBINE OIL COOLERS
3423195
INSPECT #1 BEARING ON TURBINE
3424116
Although the outage tasks have all been approved and scheduled, this REAP analysis shows how
outage scope of work in terms of cost and value can be determined, evaluated and prioritised.
1-32

1.4.5 Boiler Tube Failure Reduction and Cycle Chemistry Improvement Training
From 810 February 2000, an EPRI team presented the above training programs to a number of
plant and corporate staff. Appendix 1 lists the names of all attendees. A vertical and horizontal
cross section of staff attended, representing all disciplines operating, maintenance engineering
and management. The two-and-half day course was well received by all attendees, and they
were eager to proceed with the development of a Boiler Tube Failure Reduction program. A
plant directive Boiler Reliability Optimisation Project has been signed by the Plant Manager in
support of the program. The directive (see Appendix 2) outlines the minimum requirements to
manage a successful Boiler Reliability Optimisation program.
1.4.6 Boiler Maintenance Workstation (BMW) installation and training
During July 2000 the BMW software was successfully installed. Software user training was
given to a number of site staff.
BMW is a windows based software program used to track and trend boiler tube failure data and
information tube failures, correction and prevention, and control actions. It consists of two
modules - Boiler Works and Tube Condition. The boiler works module is a database and
graphics program designed to track maintenance failure data/information failure location and
mechanism, repair costs and methods, etc. This collated information is used to focus attention on
specific activities, for both planned and forced outages, to prevent future repeat failures. The
Tube Condition program stores and analyses data collected from various inspections, e.g. tube
wall thickness, oxide thickness data. It uses two or more data sets to determine wastage rates.
These wastage rates are used to determine remaining tube life and predict tube wall thickness.
This helps plant personnel plan future boiler tube inspections, maintenance activities and tube
replacements.
1-33

AEP Big Sandy Unit 2, Boiler Reliability Optimization - Phase 4, July 2001
Executive Summary
Utilities throughout North America are facing the challenges of deregulation and competition.
The prospect of competing in a non-regulated environment is driving utilities to find more cost
effective means to operate and maintain their plants. Recognizing this, Big Sandy Plant
participated in EPRIs Boiler Reliability Optimization Program. The project started during the
fourth quarter of 1999. The project aim was to identify improvement opportunities to attain and
sustain an availability of 95% during peak periods May through August and at least 90%
during the remainder of the year. During 2000 phase 1,2 and a portion of phase 3 were
successfully implemented resulting in sustained plant performance. As a result Big Sandy Staff
were keen to continue with the Boiler Reliability Optimization Project Phase 4, albeit with
limited funding. Phase 4 is the continuous improvement phase the proactive maintenance
aspects of the program. The emphasis is on the operations aspect that is what the operator can do
to minimize maintenance.
During July 2001 an EPRI team spent a week on site reviewing actions taken as a result of the
Phase 1 to 3, interviewing operating and maintenance staff and published a draft report. This
report documents all the findings, conclusions and recommendations accepted by Big Sandy
Plant staff.
Introduction
Utilities throughout North America are facing the challenges of deregulation and competition.
The prospect of competing in a non-regulated environment is driving utilities to find more cost
effective means to operate and maintain their plants.
Recognizing this, Big Sandy Plant participated in EPRIs Boiler Reliability Optimization
Program. The project started during the fourth quarter of 1999. During 2000 phase 2 and a
portion of phase 3 was completed with the issuing of a final report.
The staff at Big Sandy were keen to continue with the Boiler Reliability Optimization Project
Phase 4, albeit with limited funding. Phase 4 is the continuous improvement phase the living
aspects of the program.
The scope of work of this project covered the implementation of a portion of Phases 4 of EPRIs
Boiler Reliability Optimization Project and was limited to the boiler steam and water system
tubes and headers. The project consisted of the following tasks:
1. Review the progress made in implementing recommendations from previous Boiler
Reliability Optimization phases.
2. Develop a framework for specific boiler long-term plant health indicators.
3. Review and optimize/enhance the daily operation of Unit 2 to result in proactive
maintenance.
1-34

Description of Phase 4 of a Typical Boiler Reliability Optimization Project

Phase 4 is the continuous improvement - the living aspects of a Boiler Reliability
Optimization Project. It builds on the successful implementation of the previous phases and
provides proactive action and feedback on the overall boiler performance reliability and
maintainability aspects. Phase 4 focuses on the human reliability aspects as there is a direct
relationship between plant reliability and the people who operator and maintain the boiler. If the
people are reliable then the plant will be reliable. In this context human reliability is making and
keeping commitments. Phase 4 also introduces the concepts of proactive maintenance into the
world of the operator.
Proactive maintenance is an activity performed to detect and correct root cause abnormalities of
failure. It is a first line of defense against material degradation and subsequent performance
degradation, failures that ultimately lead to precipitous and catastrophic forms of failure and
plant breakdown. The operator can correct a conditional failure mode that is the cause of an
unstable condition and ensure that degradation type failures never occur. Thus proactive
maintenance is not an activity that reacts to material and/or performance type failure conditions
of a plant. Rather, it prevents such plant degradation from occurring. In reality, proactive
maintenance is a preemptive first strike against failure a true avoidance activity.
Thus, the focus of Phase 4 is on the:
role operations personnel play in the effective and efficient operation of the boiler,
tools, both hardware and software at their disposal
instrumentation to control and monitor critical component parameters and
Standard Operating instructions and procedures.
The ultimate success of a Phase 4 Boiler Reliability Optimization project is largely dependent on
the following:
Operator skills and knowledge of the boiler and the steam raising process,
Operator understanding of the effects operating practices have on both the short and longterm plant health and
The extent to which operating is involved in both the long and short-term maintenance
decision making process.
Findings
1. All of the recommendations made in the Phase 2 report have been accepted. More than 80%
have been implemented while the remaining are in the budgetary and approval processes.
Table 1-1 below summarizes the status of the actions taken and for each recommendation
given in Phase 2. A recommendation that has still to be implemented is a formalized root
cause analysis process. This is key to the successful implementation of a boiler reliability
optimization project.
1-35

Table 1-2
Phase 2 Recommendations and Actions Taken
#
Recommendations Tabled
Action taken
Specific boiler availability targets should

be developed relating to the number of
tube leaks per year. The strategy and
performance targets should be
compiled into separate plant directives,
which are reviewed annually for
effectiveness and appropriateness.
The following the best in class (BIC) goals for unit

2 have been established:
01
02
03
FOF
5.38
1.69
0.82
MOF
0.41
0.41
0.41
POF
0.0
26.85
1.92
To meet these goals a Strategic Plan for the boiler

has been compiled
2
To repair and calibrate all boiler

instrumentation critical to the efficient
and effective operation of the boiler.
This includes air, gas, water and steam
temperatures and pressures and
reheater and superheater metal
temperatures.
The installation of on-line water and

steam chemistry instrumentation and
the establishment chemical
performance index. This index can be
used in a proactive way to ensure
compliance to EPRI water and steam
chemistry guidelines.
Improved cycle chemistry monitoring with

the installation of sodium analyzer in
sample room and ChemExpert software
in the control room.
To prepare detailed inspection plans

based on the tasks identified in the
Boiler Failure Defense Plans presented
in this report. These inspection plans
should be loaded into the CMMS. This
would facilitate the development of a
scheduled and or forced outage plan at
short notice.
Prepared detailed inspection plans on

noted problems and drew up inspection
PMs.
Once the installation and the user

training on the Boiler Maintenance
Workstation (BMW) have been
completed, the BMW database needs to
be updated with all the history of
previous tube failures. This will enable
tube leaks and repairs to be tracked
and trends to be established. This will
also assist in formulating focussed
boiler inspections plans.
BMW was installed and the history data file

has been updated with data from 1994 to
2000. Currently working on updating the
history file with data from 1986 to 1994.
Fireside corrosion Confirm

mechanism and extent of wall thinning
by inspecting the area around the
burners and side wall. Also monitor O2
and CO near the tubes on the side walls
Monitoring NOx strategy has taken top priority to

measuring O2 on the side walls however
inspections and NDE is done during forced outage.
1-36
During the planned outage in 2000 the boiler tube

temperature recorders and draft gauges were
repaired. However since then some are out of
service awaiting spare parts
Chemical indices still need to be

developed.
Developed a short notice forced outage

plan.
Tasks are discussed at a weekly meeting
every Thursday morning.
Training classes for welders is scheduled

for August 2001

7
Fly ash erosion Perform a CAVT to

confirm high velocity areas. If high
velocity areas are detected design
and install baffles or flow strengtheners.
Perform a second CAVT to confirm
results and velocity profile
Only CAVT completed on the economizer and repositioned baffles and re-tested. Results not as
expected. Re-testing will continue as outage work
permits
DMW failures Replace all DMW with

Dutchmen of upgraded material.
Depending on the specific
circumstances re-positioning of the
DMW may be more cost effective
All DMW replaced.
Internal pitting Revise and update the

boiler lay-up procedure
1 Reheater lay-up procedure revised and updated.

st
Performed remaining life estimation on 1 reheater,
replaced tubes below minimum wall thickness.
Plan to replace reheater in 2002. NDE results
taken in 2000 show that there was no need to do a
chemical clean.
10
Perform a level II assessment of all final

outlet headers in accordance with EPRI
guidelines.
Completed during 2000
11
The PM Basis must be reviewed and

updated to take into account the failure
mechanisms identified during the past
outage inspection.
Developed new inspection PM to inspect repeat

mechanisms
12
The Boiler Specialist should have an

Operating Department counterpart who
would be responsible to review and
update and align operating procedures
to current practice.
As a result of the adopting the Process Centering

model, this task is now the responsibility of the
Sub-process owner.
13
Boiler tube failures/Root Cause

Analysis should be tracked and trended
BMW software will assist in tracking and trending

tube failures and their associated causes
14
The boiler inspection plans need to be

updated to take cognize of the change
in the frequency of boiler periodic
outages
The weekly outage-planning meeting discusses

and develops a forced outage plan. Inspections
plans (PM) are included in these forced outage
plans
15
A formal RCA program will result in

additional information valuable in
managing the Boiler.
st
RCA still not formalized.

Tube failures are analyzed and tracked
through BMW
16
The roles and responsibilities of the

PdM Coordinator should be clearly
defined
The PdM Coordinator position has been made

obsolete by the re-organization Process
Centering. However, the PdM activities still remain
with the previous incumbent.
17
Infrared thermography should be used

on the boiler to identify casing leaks,
valve passing etc.
Infrared thermography survey are performed

quarterly as described in EPRI Boiler reliability
Optimization IR TAD document
1-37

18
Develop an Equipment and Condition

Indicator matrix (E&CI) for all critical
boiler components. This will assist the
equipment owner to assessing the
condition of equipment under his
control.
19
To ensure a high availability between

periodic outages, 3 yearly, a detailed a
inspection, test and repair plan of all
known and potential problem areas
20
Develop a four-week rolling scheduling

process for all maintenance to assure
high utilization of resources.
It is intended to incorporate a four week rolling

schedule in the new CMMS Indus-passport
system
21
Include post maintenance testing

requirements into the planning process.
This is a must for most CM work orders.
Post maintenance testing will be incorporated into

the new CMMS
22
System owners and the Boiler

Specialist should review closeout
information for effectiveness.
The newly appointed Process Centering Subowners review closeout information.
23
Compile a management directive

identifying the minimum acceptable
standards to be complied with when
performing boiler maintenance or repair
work. This would include such
statements as every tube leak/failure
should be investigated, pad welding
should be discouraged etc.
A Boiler Reliability Optimization Directive has been

signed by the Plant Manager and is in effect.
24
Establish technology owners with

responsibilities to include condition
analysis.
Process Centering sub-owners have been

nominated in place of technology owners.
25
Need to have a more comprehensive

PDM reports as health report not
exception reports
An existing PdM health report is considered

adequate
26
Formalize monthly Boiler Condition

Based Maintenance (CBM) meetings.
Once a year, perhaps prior to any
budgeting cycle, the Boiler Specialist
should present a Boiler Failure Defense
Plan indicating the current condition,
short and long term activities/tasks that
need to be undertaken to ensure boiler
reliability.
Plan to include monthly Boiler CBM meetings into

the Boiler Reliability Optimization Directive
1-38
Trending boiler process data, using the E&CI

matrix as a base, is done using the limited
available instrumentation. Addition instrumentation
will be installed, if, and when a new DCS is
approved. It is intended to install FEGT indication
during the next outage.

27
Case histories and cost benefits

analysis of using PDM technologies
should be document either in the BMW
or PIMS databases.
Case histories are being established in the BMW

database. Metric are being developed through the
centering process
28
An annual review of the maintenance

program should be undertaken to
determine its effectiveness i.e. the right
technologies are being applied to the
right assets at the right frequency to
meet the overall business objectives
and maintenance strategy.
Included in the business strategy feedback loop.
29
Consideration should be given to having

an annual Boiler Specialist Meeting and
forming a Boiler Specialists Users
Group. This will aid in the transferring
of information and knowledge amongst
the Boiler Specialists within the AEP
organization.
AEP has formed a Boiler Reliability Optimization

Team whose responsibility is to support the
implementation of the program
30
Consider providing Level of Awareness

training (LOA) training for the entire
staff on Plant Maintenance
Optimization.
LOA training has been done previously. Not

considered necessary at this stage.
31
Re-train all PIMS users on the details of

PIMS and how best to it
PIMS is to be replaced by Indus passport by

January 2002. Training on the new system starts
in the last quarter of 2001
32
Develop a series of inspection tasks

using the data and information identified
and update the CMMS.
Inspection tasks for 2000 have been completed.
2. Five operators with a wide range of experience were interviewed (four Unit operators and
one Equipment Operator). The interviews focussed on the knowledge of the boiler and the
steam and water cycle. Responses to questions from some of the operators regarding boiler
operation and operating parameters on critical equipment were incorrect. Critical components
within the boiler were not known. Under base or transient loads the focus is on steam
temperature only, metal temperatures are not considered. These same operators have been
given latitude to develop their personal operating practices and to experiment with set points
on critical equipment. The combined effect of the above can lead to an increased risk with
the consequent reduction in unit performance, both efficiency and reliability.
3. Each unit operator interviewed uses his or her own set of procedures - start-up procedure,
marked up with his or her own notes. The start-up procedure reviewed in the control room
was dated 1970, where as, the set given at the beginning of this project was dated October
1998. Not all operators were aware there was a new/revised start-up procedure. Those who
had a revised copy preferred to use their own marked up copy. This results in inconsistent
start-up times, and possible inappropriate actions during unit start-ups.
1-39

Operating instructions in the form of unsigned, undated, hand written notes were found taped
to the control panel. The operator bases identification of the author of these instructions on
trying to recognize the handwriting. Operators have no way of knowing what new operating
instructions have been issued unless they see a new piece of paper taped on the board or page
through the log book assuming it was written. There is no way of ensuring that all operating
personnel are being made aware of changes in critical operating parameters.
4. There is no documentation standard or control of Operating Procedures and Instructions.
Procedures reviewed did not follow a common format, approval signatures, date issued,
revision number etc. The new Unit Lay-up procedure given for review was in the form of an
email.
5. Operators are not always informed of equipment, including chart recorders, being taken out
of service for maintenance work. Unanticipated requests to have equipment taken out of
service and clearances issued results in inefficiencies and delays for both maintenance and
operators. The introduction of the Forced Outage Maintenance Work matrix posted in the
control room has eased this situation. The matrix is an excellent communication tool. It
keeps operating personnel informed of what work will be done under several forced outage
scenarios. This puts the operators in a proactive mode regarding clearances and equipment
shut down priorities.
6. Experienced and qualified Laboratory personnel control unit chemistry within EPRI
guidelines. Operating personnel have limited knowledge of unit chemistry or the effects of
operating outside EPRI guidelines on unit reliability. Operators are in a reactionary mode
regarding chemistry excursions, depending on laboratory personnel for direction and
corrective action.
There are no feedwater or boiler pH indications in the control room. On a once-through,
super critical unit using an oxygenated treatment (OT), maintaining a proper pH level is of
the highest importance, as there is only a short time window of opportunity to react to low
pH levels. Lack of pH readings in the control, coupled with the limited knowledge of
operating, presents an increased risk to the reliability of the unit.
7. Operators spend approximately 25% of their time taking manual readings for staff personnel.
A number of these reside in the computers in the control. The need for and the information
from these readings has not been explained and or justified. This leads to resentment and
frustration between operators and other staff members.
Conclusions
1. Big Sandy staff needs to be congratulated on the effort they have expended in successfully
implementing the recommendations from Phase 2 Boiler Reliability Optimization Report.
This effort shows the commitment to improve and sustain high performance.
2. There is a lack of management control in the operations area. Operators are not held
accountable for their individual actions, nor the results of their actions on Unit efficiency or
reliability.
1-40

3. Operators are not being given the tools or resources they need to operate the Unit in a
consistently safe, reliable, and efficient manner. Operating procedures and instructions are
written and distributed in a uncontrolled manner. Training of operators is inadequate and
inconsistent. As a result, there are no primary standards for operation or control of critical
Unit processes.
4. The loss of many older, more experienced operators and supervisors has diluted the
knowledge base of the operations group. This has resulted in a wide range of operating
philosophies among the Unit Operators.
5. Chemistry control is good due primarily to the efforts of the laboratory group personnel.
Operations contribution to the success of this program is minimal due to their lack of
knowledge of the chemical processes involved.
6. Control room instrumentation is an extension of the Unit Operators senses. The sense of
urgency related to instrument repair and/or replacement needs to be raised to a new level.
This should be accomplished in the near future if the practice of assigning only one operator
to the control room is to be successful.
7. If Big Sandy is to successfully implement a Pro-Active Maintenance (PAM) program, it is
essential that the Operations group plays an active role with its implementation. Without
their full commitment, success will be difficult. Serious consideration should be given to the
preceding recommendations before plant management attempts to implement such a
program.
Recommendations
1. Continue with the efforts to implement the outstanding recommendations from Phase 1
(above table) especially with installation of a FEGT monitoring system. This will allow the
operator to load the unit to its maximum without placing excessive thermal stresses on the
super heater.
2. Develop a series of long term plant health indices for the boiler: thermal excursion, chemical
and trip index.
Thermal excursion index is the number of additional equivalent operating hours
experienced by the most sensitive header in the boiler as a result of metal temperature
excursions in excess of design temperature with a boiler pressure > 80% of normal
o
operating pressure. In addition, all excursions in excess of design + 100 F shall be
included irrespective of boiler pressure.
The Additional Operating hours = Constant (k) x ( T x t)
Where:
T = difference between the peak temperature reached in an excursion and the design operating
metal temperature
t = total duration of the excursion in hours
1-41

Constant (k) = approximation, assuming a straight-line relationship between elevated

temperatures and equivalent operating hours.
Thermal index = Additional Operating hours x Total hours in month divided of unit operating
hours for the month.
Chemical Excursion Index is a set of indices in which measurements taken on the
plant of chemical conditions of steam, feedwater, etc are normalized to a base of one.
The actual values are manipulated mathematically to give a result that is of the order
of one. A result of 0.5 is excellent and greater than 0.9 is considered unacceptable.
Results of between 0.6 and 0.9 indicate chemical condition under control and no risk
to long term health. Results between 0.9 and I are acceptable for short duration only.
Results greater than 1 require immediate attention. These results can be represented
in a bar chart and displayed to the operator
Chemical index = of all incidents of unit parameters > 0.9 for current month x Total
hours in month divided by of operating hours for the current month
Trip index is the accumulative total number of automatic and manual trips of the unit.
This will give a feel for the number of stress cycles the plant has been subjected to no
matter what was the cause
Trip Index = of all trips for current month x Total hours in the month divided by sum of unit
operating hours for the month
These indices can be developed for all units and used as a means to compare unit
performance
Note the numbers quoted above are guidelines, they should be adjusted to suite the
particular unit circumstances and conditions
3. Analyze the effects of the wide range of conditions under which the Unit is operated. Select
the Best Practice and develop a set of procedures and hold all Unit Operators to those
standards. The start-up procedure should have target times for mile stone activities and check
boxes for the operator to insert the actual time and reasons for any deviation from target.
After the start-up these times can be analyzed for improvement opportunities.
4. Operations group should be involved in planning and scheduling of maintenance on any
equipment that needs to be taken out of service. Operators should be informed on the status
of any instrumentation that is out of service. They should know why, and the expected return
to service date.
5. Develop a document control system and conduct periodic compliance audits to ensure the
correct document/procedures are being used. This system should include all operating
procedures and instructions. As a minimum, these should include signed approval levels,
revision date, distribution list, standardized format, and collection and disposal of outdated
documents. A list of typical headings found in a procedure can be found in Appendix 2. One
complete set of operating procedures and a temporary operating instruction book should
reside in the control room. All entries in this book should be signed and dated. Operators at
the beginning of their shift should review this book for updated changes.
1-42

6. Provide training to all operating personnel on basic unit water and steam chemistry and the
alarm and action level guidelines.
7. Provide pH indication for condensate, economizer inlet, and main steam in the control room.
8. Review the need and frequency for the manually recorded readings. Values, which reside in
the computer, should be collected automatically by creating a specific computer log. The use
of and value obtained from the readings collected manually should be feed back to the
operators.
9. Develop and formalize an improvement program - root cause analysis process. Use the
following as a guide.
Obtain Management support such a program financial and human resources
Select and nominate a champion owner of the process
Review the industry and select a root cause analyses system using the following
seven factors:
Easy to use in the field by non-experts
Effective in consistently identifying root causes
Well documented
Accompanied by effective user training
Credible with the workforce
Helpful in presenting results to management
Designed to allow collection, comparison, and measurement of root cause
trends.
Train investigators/ Coaches representation from Operating and Maintenance
Give awareness training to operators and craftsperson
Compile an administrative procedure purpose, objective, scope responsibilities,
investigation and reporting matrix etc
Develop a database for causes and corrective actions
10. Provide training to ALL operating personnel on Unit chemistry guidelines and emergency
responses. Of special importance is the relationship between specific conductivity, ammonia
levels, and pH values in the condensate/feedwater systems. The ammonia feed-rate is
controlled using the specific conductivity of the feedwater, which maintains the target pH
values. The relationship between conductivity values and ammonia is a straight-line
relationship, whereas the pH is logarithmic. Changes in conductivity are the operators first
indication of approaching problems. Operators should also understand the difference between
cation and specific conductivity, as well as the importance of sodium levels. Provide Ph
reporting instrumentation to the control room. This should pH values on the condensate,
feedwater, boiler, and main steam.
1-43

Appendix 1: Interview List
1-44
Name
Designation
D. Mell
Production Support Manager
J. Burton
Region 3 Engineer
E. Dillow
Region 3 Engineer
A. Robinson
Production Services Leader
S. Jenks
J. Skaggs
Lab Technician
D. Pinson
Unit Operator
R. Lewis
Unit Operator
J. Adkins
Unit Operator
10
R. Peck
Unit Operator
11
M. Keene
Equipment Operator
12
M. R. Mckenzie

Appendix 2: List of Typical headings that appear in a policy and or a

procedure
1. Group name
2. Document type: policy, standard, guideline, procedure etc
3. Reference number
4. Revision number
5. Description of revisions made
6. Review date
7. Date authorized
8. Date when document came into effect
9. Compiled by
10. List of designations who have approved the document
11. Authorized by
12. Title
13. Background/preamble/introduction
14. Purpose
15. Scope of document
16. References
17. Definitions and or abbreviations
18. Responsibilities
19. Responsibility matrix
20. Requirements: step by step of what is required to be done
21. Work process flow and responsibility matrix
22. Distribution list
23. Appendices
1-45
2
PHASE 2 BOILER INSPECTION PLAN REVIEW AT
GREAT RIVER ENERGYS COAL CREEK STATION
UNIT 2
2.1 Executive Summary

Senior Management made the decision to change the boiler maintenance overhaul frequency
from two to three years, and wanted assurance prior to the Spring 2001 outage that the existing
boiler inspection plans were comprehensive and complete. Confidence in the newly adopted
maintenance strategy is essential, as the next general overall outage after the Spring 2001 outage
is in 2004. EPRI was requested to review the current Unit 2 outage inspection plan and
recommend changes and/or additions, to provide this assurance.
During the week of August 21, 2000, two EPRI representatives interviewed a number of Coal
Creek Power Plant staff members from the operations, maintenance, engineering and
management groups. The unit history, maintenance, operating practices, and the boiler
inspection plan were reviewed and analyzed.
The current boiler inspection plan is reactive, i.e. it focuses almost entirely on known problem
area sootblower erosion - and does not cover all the components within the boiler in sufficient
detail. There is a limited predictive element to the plan. It does not address any future or
potential failure mechanisms, and is therefore not comprehensive and will not give the necessary
assurance that all risk areas have been adequately covered.
This report highlights a number of recommendations that would assist in developing a
comprehensive and effective boiler inspection plan. These recommendations, together with
comments/suggestions added to the Unit 2 Outage Inspection Plan given to the EPRI team on 21
August 2000, will provide a quality inspection plan. Annexure II
An extract from another utilitys boiler inspection plan and is given in Annexure I to illustrate
the detail that is required in a typical Boiler Inspection Plan.
2.2 Introduction
Faced with de-regulation and competition, Senior Management at Great River Energy Coal
Creek Station, challenged the existing plant maintenance strategy and sought opportunities to
improve plant availability and reliability. One of the changes made was to the overall unit
2-1

Boiler Reliability Optimization Project Phase 2 Boiler Inspection Plan Review at Great River Energys Coal
Creek Station Unit 2
maintenance strategy, i.e. the frequency and duration of outages was changed from 2 to 3 year
intervals and a shortened duration. This change would improve the overall unit availability. To
improve unit reliability the focus shifted to understanding and eliminating the causes of boiler
forced outages; the predominant cause being boiler tube leaks. To ensure no surprises a
comprehensive boiler inspection plan needs to be developed and implemented during the next
outage Spring 2001 .
A boiler internal inspection is one of the most important inspections to be carried out during an
outage. It must be done timeously, systematically, accurately and by experienced inspectors.
The quality of this inspection will ensure that a reliable boiler will be put back into service. The
only approach is do it right the first time all the time
The selection and number of inspectors should not be based on quantity but rather on quality.
The staff chosen must be committed to the assignment and have sufficient experience and
background to identify any abnormalities. The staff should have a good knowledge of the gas
flow patterns, temperature profiles, material composition and movement/expansion of the boiler.
EPRI was requested to review the existing plans and suggest changes and or additions that will
give this assurance, to provide assurance that an adequate and comprehensive plan was in place.
During the week August 21, 2000 two EPRI representatives visited the plant, reviewed and
discussed the current boiler inspection plan and tube leak history data with the boiler engineer,
and interviewed a number of staff from the operating, maintenance and engineering groups.
2.3 Findings and Observations

The findings below are based on the Unit 2 outage inspection plan as presented during the review
(see Annexure II) and on the personal interviews.
2.3.1 The inspection plan is divided into five sections: burner front, waterwall, Superheater
and reheater, back -pass and boiler external. Although the boiler as a whole is covered,
the specific inspection details of the areas within each section are not covered sufficiently
e.g. superheater, DMW etc.
2.3.2 Each section has been assigned to a responsible person who will manage the inspection
and data collection. The boiler engineer has overall responsibility to ensure adequacy of
inspections, data collection and coordination between the various sections.
2.3.3 The plan is reactive and focuses almost entirely on inspecting known problem areas e.g.
identifying and correcting soot blower erosion damage.
2.3.4 Areas within each of the five sections are not prioritized and no go no go criteria
given e.g. minimum tube wall thickness and crack/groove size/length and shape.
2-2

2.3.5 Documentation of the pre-outage external inspection and walk-down of the boiler is
comprehensive
2.3.6 Infrared survey of the boiler casing and valves is not included in the pre-outage
inspection.
2.3.7 No provision is made to complete a dirty boiler inspection.
2.3.8 The present sootblower maintenance strategy is not in line with current or future
operating mode of the unit. Sootblower preventive maintenance has not been adequate,
as shown by the number of corrective maintenance activities that have needed to be
performed. The top five SB problem areas are: Feed tube wear, Sprocket and chain
wear, Bearing failures, Packing failure and Carriage wear
2.3.9 Thermal drains are being replaced with steam traps without a thorough root causes
analysis being done to ensure that this is the correct action to take. There is a limited
history regarding failures and repairs.
2.3.10 Result trending and predictions cannot be adequately completed, as there is no central
database to record and keep boiler test and inspection data and information. Boiler
inspection reports that identify actions taken, future actions needed, or lessons learned,
are inadequate. All history data on the boilers tube leaks is owned and maintained by
the Boiler Engineer and is in hard copy format.
2.3.11 Root Cause Analysis is not formally done. The Boiler Engineer maintains tube leak
reports, using CE Availability data sheets. Not all tube leaks are investigated. History
summary is available from 1989. The summary includes date, suspected failure cause,
corrective action taken, and recommended outage follow up actions.
2.3.12 Startup and shutdown procedures are combined. They are too general and give little
guidance as far as criteria to be met metal temperatures have been exceeded in the past
because of this. No precautionary measures are given to the operator, e.g. when firing the
boiler at over pressure.
2.4 Recommendations
2.4.1 A more detailed proactive inspection plan is needed to support a three-year outage cycle
plan, while meeting plant goals. The boiler components are time and temperature
dependent. Over-stressing these components will accelerate aging. By adopting a
proactive stance, knowing the remaining life of these components, replacement plans can
be developed and implemented before aging/failure affects unit reliability.
2.4.2 A schedule should be developed for all inspection areas in the boiler. Prioritization
should be based on potential findings that would affect outage duration, safety, budget
restraints, and unit reliability. All areas that have a high priority should be completed by
the end of the first week. This requires a pre-outage study of the duration and cost of
inspection techniques to be used. It should include component disassembly, scaffolding,
2-3

insulation removal, surface preparation, ventilation, lighting, and environmental

compliance.
2.4.3 Criteria need to be developed for the inspection of each area, based on historical data,
possible damage mechanisms, typical locations, inspection techniques, required
preparation and support, and estimated inspection times. It is important to establish a
definition of what are permissible flaws, and what action should be taken regarding a
repair/replace decision when an unacceptable flaw is found. Non-critical flaws should be
duly noted, entered into the database, and trended over at least two outage cycles.
2.4.4 All data gathered during the inspection, as well as all repair/replace decisions, should be
entered into a database maintained by the Boiler Engineer. This database should include
as found and as left conditions, repairs made, and recommendations for future
outages. Examples of both inspection and repair record formats are included in the
annexure I
2.4.5 Expand inspections of the superheater and reheater sections beyond soot blower damage,
broken clips, rubbing damage, and ash erosion to include remaining life assessments. UT
measurements should be taken to determine the oxide thickness on the superheater.
These values can be used to determine the high temperature areas in the SH and will
validate the temperature profile as measured by the installed tube metal temperature
instrumentation. Tube samples should be taken from hot areas of the SH for metallurgical
analysis to determine remaining life. These analyses may have an impact on operating
practices.
2.4.6 Horizontal and pendant SH sections are difficult to drain completely, if at all. As a result,
they often suffer internal damage due to pitting from condensation. Tube samples should
be taken and analyzed for this.
2.4.7 Header base line data, internal and external, needs to be obtained, as it would indicate the
effect of past operating modes, and provide information to determine future operating
modes. Superheat and reheat headers have never been inspected. Fiber optics can be
used to determine if any erosion or cracks have developed.
2.4.8 The current inspection plan calls for inspection of the remaining wall blower refractory
tabs. Some of these tabs have been removed, as they have been a source of tube leaks.
The remaining tabs should be removed, as they are neither used nor needed and are
potential areas for tube leaks.
2.4.9 High stress raisers are potential sources for tube leaks. Inspect and itemize all internal
boiler tube attachments for cracks, erosion or damage.
2.4.10 An infrared survey of the boiler casing, valves, and other potential sources of heat loss,
should be performed as part of a pre-outage inspection.
2.4.11 A dirty boiler inspection should be included in the boiler inspection plan. Many will
argue that this inspection is uneconomical. However, there is much to be gained by
2-4

observation and correct interpretation of what is observed. An experienced boiler

engineer can benefit enormously and gather insight of what is going on inside the furnace
from this inspection. Visually examining ash deposits color, size, build up and profiles
- and tube surface conditions, will highlight areas of erosion and high or low gas flow
regions.
2.4.12 Although no Dissimilar Metal Weld (DMW) failures have been reported, welds should be
inspected visually at every outage using UT. DMW tube samples should be taken and
analyzed for remaining life.
2.4.13 The sootblowing systems design should be reviewed. This should include an evaluation
of reliability, availability, maintenance and operating issues. This review should also
include an evaluation of the effectiveness of steam traps in eliminating water in the
system. From this design review an optimized sootblower operation and maintenance
strategy can be developed, implemented, and issued regularly via the Maximo system.
2.4.14 Perform a functional check on all sootblowers. Each sootblower should be operated in
turn and checked visually from the inside of the furnace for correct nozzle position,
blower insertion length and rotation.
2.4.15 Full implementation of the Maximo system should include the following.
All identified boiler inspection tasks should be kept in the Maximo system as
preventive maintenance (PM).
These PM should include, but not be limited to, duration, location, man-hours,
equipment needed, and collaboration needed with other groups.
2.4.16 Implement EPRIs Boiler Maintenance Workstation (BMW). BMW is a Windows-based
software program used to track and trend boiler tube failure data and information tube
failures, correction and prevention, and control actions. It consists of two modules Boiler Works and Tube Condition. The boiler works module is a database and graphics
program designed to track maintenance failure data/information failure location and
mechanism, repair costs and methods, etc. This collated information is used to focus
attention on specific activities, for both planned and forced outages, to prevent future
repeat failures. The Tube Condition module stores and analyzes data collected from
various inspections, e.g. tube wall thickness, oxide thickness data. It uses two or more
data sets to determine wastage rates. These wastage rates are used to determine
remaining tube life and predict tube wall thickness. This helps plant personnel plan
future boiler tube inspections, maintenance activities and tube replacements.
2.4.17 Following every planned outage, a detailed report should be issued. This report should
identify inspection results, repairs and modifications, lessons learned, and activities that
need to be planned for the next outage.
2.4.18 All NDE data on boiler pressure parts should be trended and used for failure prediction
and long term planning.
2-5

2.4.19 Senior Management should support a formalized Boiler Tube Failure Reduction team.
The team, headed by the Boiler Engineer, should have members from the operations,
maintenance and engineering groups. The team should follow a recognized root cause
analysis process when analyzing all boiler tube leaks. Records should be kept in a central
database and failure statistics and tube condition minimum wall thickness, oxide
thickness and remaining life results, etc., should be tracked and trended.
2.4.20 Separate Unit Start-up and Shut Down procedures should be written, as these are two
distinctly different activities each with specific criteria and operator requirements. Each
procedure should identify the precautions to be taken before or during each step. This
should be followed by a step by step sequence of actions with criteria to be met, e.g.
metal temperature rates of change, pressure ramp rate, differential temperatures, etc.
Each action should have a check box and a remark column where the operator can check
off the action and explain/give reasons for any deviation from the specified criteria.
2-6

ANNEXURE I: Extract from a Boiler Inspection Report

This report covers all boiler internal inspections, including tubes, burner tips, and dead air spaces
performed by Plant personnel. Inspections of the mud drums, the main steam drum, and
penthouse are included.
1st digit (location):
1
2
3
4
5
Firebox
Upper Arch
Down draft
Dead-air Spaces, Drums
Penthouse
2nd digit (tube type):
1
2
3
4
5
Economizer Tubes
Waterwall Circuit Tubes
Steam Cooled Tubes
Reheater Circuit Tubes
Superheater Circuit Tubes
Inspection Areas
Most of these are known areas where failures have occurred or repairs have been made during
past Periodic Outages and Forced Outages. DM designates D-meter inspections for tube
thickness.
1200 Waterwall Cleaning Devices, operational condition

This inspection involves both pre-outage and outage responsibilities. Operations must check
blower and lance operating conditions prior to Unit shutdown. This requires that all water-lances
be tested for proper speed settings and visually verified for operating effectiveness, noting
condition of the booster pumps and blowing pressures. All "partial-arc" wall-blowers must be
verified for proper operation. Note any existing conditions that may effect tube life and closely
inspect theses areas (see item Nos. 1209 and 1210). A furnace side inspection is to be performed
by the Boiler Inspection Team from a furnace scaffold, including nozzle head conditions and
insertion depth settings on all wall blowers.
Inspection results
All wall blowers were inserted for inspection and checked for appropriate depth setting, yielding
20 required adjustments (vs. 43 total in 1996). Lists of all damage and needed adjustments were
given to the Unit Planner. Depth setting adjustments were made during the outage, while the
lance changes were scheduled for after the outage due to parts availability. Over half of the
original water-lances (WL) were found with damaged lance tips (cracking or holes).. Lists of all
damage and needed adjustments were given to the No. 7 Unit Planner
2-7

Recommendations for 2004:

Several of the lances were binding on start-up, and at least three of the lance tubes will need
replacement. Air cooling to the control boxes will be added to keep the box temperatures below
0
122 F. Access to WL nozzles inside the windbox will need to be made during a furnace cleaning
outage. For 2004, wall blower and WL repairs and adjustments are expected to be minimal.
1501 Superheat Division Panels, attachment problems
Look for misalign tubes and possible resulting damage. Look for failed "wrapper tube" ties.
Look for abrasion damage from the attachment ears rubbing into adjacent tubes. Recommend
repairs.
division
tub
attachment
rub damage
Attachment ear
Figure No.
area(s)
Inspection results:
There were 10 locations found and repaired (vs. 31 found and 23 completed in 1996). Of these,
two were failed attachments, and 8 were from attachment rubbing damage. Some of the rub
damage locations were severe, causing abrasion damage in excess of 1/2 wall thickness (see Fig.
6). The recommended repair for the ear rubbing was to remove that portion of the attachments.
Recommendations for 2004:
Continue inspections for failed attachments and attachment ear rubbing. Expect nearly the same
amount of work as in 2001.
2-8

ANNEXURE II: Unit 2 Outage Inspection Plan as of August 21, 2000

This plan contains numerous visual checks and does not give guidance on what to look for, how
to look for it, and what to do with the data/information obtained? Some guidance needs to be
documented for each team on what to do if cracks or dents are identified, e.g. measure wall
thickness, cut out and insert a Dutchman, weld over lay, etc.
Comments/additions have been made in each section below. For ease of identification they have
been underlined.
1. Burner Front Team Member
Will perform burner front inspection and coordinate all repairs of the burner from inside and
outside the boiler. The following inspections will be performed:
1.1 Prior to the outage (within 2 weeks of the start of the outage)
Conduct and external walk down of the burner fronts from 5th floor to 10th. The items to check
shall include:
Look for any coal or ash leaks around the coal piping and nozzles.
Check the condition of all coal piping spring hangers.

A. Record the hot position of the hangers.
B. Check if any hangers are bottomed out.
C. Visually inspect the hanger rods and attachment pins.
D. Inspect the hanger lugs on the support steel and on the coal piping.
Note if any hagen drives are disconnected.
Inspect the air supply and signal lines to the hagen drives. Check for any broken or
disconnected lines and to see if any lines have broken loose from their supports. Check the
condition of the multi tube lines from the burner front to the LIE cabinets.
Check with Operations and the E&I techs to see if any of the tilt drives are short stroked.
Visually inspect the external tilt linkage for any members that are bent, broken or
disconnected.
Note from the Honeywell any flame scanner with low intensity readings. These scanners
will be inspected in detail after the unit is off line.
Check with the E&I techs to find out if there are any flame scanners that they have had
trouble inserting or removing. Also check to see if there are any work orders on the flame
scanners.
Check for any oil leaks around the burner front oil guns and supply piping.
2-9

Inspect all of the oil guns and spark igniters and oil gun air lines.
Visually inspect the new SAS air ducts on 9th and 10th floor.
A. Visually inspect the SAS duct hanger rods and attachment pins.
B. Check for any interference's between the SAS registers and the structural steel.
C. Inspect the windbox hanger rods on 9th floor.
Visually inspect the fabric expansion joints.
Inspect all external pipe work of the various systems for steam and water leaks, sagging,
abnormal internal noise, signs of shock, pipe guides, hangar supports free movement of
constant and spring supports, missing bolts and nuts and dust covers, signs of corrosion and
integrity of the supports.
Inspect boiler skin casing cladding for damage, missing or corroded sheets.
Inspect all ducts for air and gas leaks, abnormal movement or vibration, defective damper
linkages or fouling with moving parts.
Inspect the boiler main and sub structure for deformation, missing bolts and nuts, flame
cutting damage, severe corrosion, signs of unauthorized welding on main load bearing
beams, beams/buckstays for sagging and or distortion.
1.2 At the start of the outage (During the installation of the boiler scaffolding)
Inspect the windbox and windbox expansion joints.
Stroke all of the windbox dampers manually, checking for any that are binding and that the
connector link between the damper blades is intact.
Remove all the flame scanners and insert an old one in each guide tube noting any problems
with inserting the flame scanner.
Manually stroke the tilts on the over and underfire air compartments and the underfire air
compartment dampers.
Stroke all aux air and SAS yaws.
Inspect the inside of the SAS ducts. Check condition of the fabric expansion joints and air
dampers.
1.3 Burner Front inspection (Upon completion of the boiler scaffolding)
Insert all oil guns and check the dimension from the face of the diffuser to the oil gun tip.
See the CE drawing for details.
Insert all of the ignitors and check the insertion depth and position of the ignitor tip.
Inspect the oil nozzles looking for the following items.
2-10

A. Condition of the tip. Look for distortion and burning of the tip.
(Oil guns should be tested in a test rig to ensure proper oil spray angles. This help
determine the performance of each gun)
B. Check condition of the oil gun and the flame scanner guide tubes.
C. Look for broken or bent tilt linkage. Note if any lever arms are working off of the
stationary pivot pin.
D. Note if any of the diffuser discs are missing on the elevation 5/6 and 7/8 oil
nozzles.
Inspect the coal nozzles looking for the following items.

A. Condition of the two piece tip. Determine which tips will be repaired or replaced.
B. Check for thinning of the removable transition piece on the front of the coal
nozzle body.
C. Check for missing or loose ceramic tile in the coal nozzle body and gate body.
D. Look for broken or bent tilt linkage. Note if any lever arms are working off of the
Inspect the aux air and SAS nozzles looking for the following items.
A. Condition of the tip. Look for distortion and burning of the tip
B. Check condition of the yaw linkage, looking for any bent or broken pieces.
C. Check condition of the scanner guide tubes in the aux air nozzles.
D. Look for broken or bent tilt linkage. Note if any lever arms are working off of the
Use the form shown as Attachment 1 to record all of the repairs that the contractor shall perform.
All repairs such as nozzle replacement, linkage repairs and nozzle ceramic repairs should be
detailed on the form.
2-11

2. Waterwall Team Member

Will perform inspections and coordinate repairs of the waterwalls, center wall and slopes. The
following inspections will be performed.
2.1 Prior to the outage (within 2 weeks of start of the outage)
Review the tube leak records for the last three years and identify any areas that need further
repairs. This should include any areas where window were installed or any areas where tubes
were just welded shut.
2.2 Prior to the installation of the boiler scaffolding (after the throat scaffolding is
installed)
Perform a general visual inspection of the slope before the scaffold is installed. Check for
large dents in the slope, dented tubes and worn areas.
The wear areas should include the following.
A. The slope along the side walls under corners 2 and 7.
B. Both the front and rear slope along both sides of the center wall.
C. The side walls at the throat opening.
D. The center wall where it penetrates the slope.
2.3 Upon completion of the boiler scaffold
Clean the waterwalls and perform an UT inspection of the waterwalls between the upper
burner front and the SAS registers. See the attached drawing, Attachment 2 for details of the
area to be inspected.
Inspect all of the wall blower openings. The following items should be checked.
A. Visually inspect for any worn or eroded tubes. Mark any tubes that may need a
follow up UT inspection. Mark any tubes that require replacement.
(Criteria need to be developed 70 % wall thickness is commonly used)
B. Check the condition of the refractory and mark any openings that require
refractory repairs. A list of the openings requiring refractory repairs should be
forwarded to the External Boiler Team Member.
C. Visually inspect for any cracking occurring at the tabs welded around the
opening. (Remove unnecessary tabs by carefully grinding)
Inspect the crotch tubes at the top and bottom of the burner front and SAS register openings.
2-12

Perform an UT inspection of the boiler nose. (Go and no go criteria need to be

provided). See the attached drawing, Attachment 2 for details of the area to be inspected.
Inspect the bends in the waterwall tubing below the nose. Perform a random UT inspection
of the tube bends. (record the measurement position and tube location as a reference)
Inspect the bends at the bottom of the radiant reheat section. Perform a random UT
inspection of the bends. Also check for any tubes out of alignment or bent tubes.
Visually inspect the joints between the center wall panels. Check for rubbing damage on the
tubes at the panel joints.
Perform a visual and UT inspection of the slope looking at the following areas.
A. The front and rear slope along both sides of the center wall. The area to be
inspected should be the full length of the slope for a 20 tube wide panel.
B. The slope along the side walls under corners 2 and 7. The area to be inspected
should be the full length of the slope for a 20 tube wide panel.
C. The slope just below the upper bends directly under burner fronts 2, 4, 5 and 7.
Inspect the front wall between the burner front and radiant reheat for any tubes that are
smashed or dented.
Replace the lower radiant reheat bend and section of tubing in furnace A that previously
failed.
Remove a two tube samples from the waterwalls. The sample should be removed from the
front or rear wall on ninth floor between the burner fronts. (consider taking more samples to
get a representative set)
2.4 Upon completion of the bottom ash scaffolding
Visually inspect for any smashed or dented tubes at the throat opening.
Check the side wall tubes at the throat opening. Check the horizontal tubes at the center joint
of the side wall.
Inspect the lower side of the throat tubes.
2-13

3. Superheat and Reheat Team Member

Will perform inspections and coordinate repairs of the superheat and reheat pendants. This work
will be completed on night shift. The following inspections will be performed.
repairs. This should include any areas where windows were installed or any areas where
tubes were just welded shut.
3.2 During the outage
Visually inspect all of the sootblower lanes on the leading edge of the superheat pendants.
This work will be performed using the boiler scaffolding.
Visually inspect the fluid cooled spacer tube that runs through the front of the superheat
pendant. Check for broken clips or rubbing damage.
Scaffold the sootblower lanes in the center of the superheat pendants and visually inspect all
sootblower lanes.
A. Special attention should be given to inspecting the double offset bends in the
crossover area.
B. Visually inspect the fluid cooled spacer tube that runs along the center wall.
Check for rubbing damage.
C. Check for any broken or missing flex connectors in the pendants
D. Inspect the bent tubes located on the A side of the furnace. Determine what
repairs will be required to straighten the tubes.
Scaffold the sootblower lanes between the rear waterwall hanger tubes and the reheat
pendants. The following inspections should be performed.
A. Perform a UT inspection of the trailing edge of the rear waterwall hanger tubes at
each sootblower lane.
B. Inspect the rear waterwall hanger tubes where they penetrate the slope. Check for
ash erosion on the trailing edge of the tube from ash sliding down the slope and
check the leading edge for flyash erosion.
C. Visually inspect the leading edge of the reheat pendants and the trailing edge of
the superheat pendants at each sootblower lane, checking for sootblower erosion.
D. Visually inspect the fluid cooled spacer tubes located in the center of the boiler on
the back side of the superheat pendants. Check for rubbing damage between the
spacer tube and the rear waterwall hanger tubes and superheat pendants.
2-14

E. Inspect the fluid cooled spacer tube that runs through the front of the reheat
pendants. Check for broken clips or rubbing damage.
F. Inspect the front lower bends of the reheat pendants. Check for sootblower
erosion on the inside and outside of the bends and for any missing or damaged
tube shields.
Perform oxide thickness measurement on the hottest across the superheater to determine
the temperature profile.
Remove samples for metallurgical analysis creep damage and remaining life.
Inspect DMW for cracks. If cracks are found cut out a section and replace with a
Dutchman. If no cracks found cut out a section from the hottest area for metallurgical
analysis.
Scaffold the sootblower lane on the back side of the reheat pendants. The followings
inspections should be performed.
A. Visually inspect the sootblower lane looking for sootblower erosion. Special
attention should be directed to the double offset bends in the crossover and to the
bottom side of the horizontal crossover tubes.
B. Visually inspect the spacer strap in the reheat finishing section, looking for
broken or missing U straps.
C. Check for any broken or missing flex connectors
D. Inspect the tube shields at the bottom sootblower on the trailing edge of the
reheat pendants. Check for missing or damaged shields.
Inspect all sootblower openings along the side wall. Check for tube erosion and for cracking
along the tabs. Also visually check the condition of the lance nozzles.
Inspect the roof and slope tubes, checking for bent or warped tubes.
When performing pendant inspections special attention should be given to the first five
pendants from each side wall. Historically the first five pendants have suffered more severe
sootblower erosion.
2-15

4. Back-pass Team Member

Will perform inspections and coordinate repairs of the back-pass from the furnace screen tubes to
the economizer. The following inspections will be performed.
repairs. This should include any areas where windows were installed or any areas where
tubes were just welded shut.
Perform an UT inspection of the inner radius of the lower bend of the furnace screen tubes
just above the refractory dam.
Visually inspect the spacer straps in the front of the superheat finishing section looking for
missing or broken U clips. Also inspect the spacer straps that wrap around each pendant.
Check for any broken or cracked welds on the front of the wrap around straps. 30 new straps
have been ordered based on a previous inspection. Inspect for sootblower erosion along the
lower sootblower lane on the front of the superheat finishing section and the rear of the
furnace screen tubes.
Scaffold the sootblower lanes on the front side of the superheat finishing section. Visually
inspect the condition of the tube shields on the trailing edge of the furnace screen tubes.
Visually inspect the condition of the superheat pendants checking for sootblower erosion and
broken or missing flex connectors.
Visually inspect the rear extended backpass floor tube located directly in front of the
superheat backpass front tubes. Check for sootblower erosion on the floor tube that occurs in
the spaces between the backpass front tubes.
Scaffold the upper sootblower lane on the back side of the superheat finishing section.
Check for sootblower erosion on the trailing edge of the superheat pendants and on the
leading edge of the backpass front wall tubes.
Visually inspect the spacer straps located on the backpass front wall tubes.
Check for any U clips that are broken or have burned off. Also check for any
damaged tube shields in this area.
The sootblower lane for retracts 19 and 48 which is located at the rear of the
front wall tubes will require several inspections.
missing or
superheat
A. The trailing edge of the superheat front wall tubes and the top side of the
horizontal superheat tubes should have a UT inspection performed on them along
this sootblower lane.
B. The front side of the superheat low temperature pendants should be visually
inspected for sootblower erosion and for missing or damaged tube shields.
Special attention should be given to the lower bends on the low temp pendant
section.
2-16

C. There was a previous tube failure on the West Side on the top of the superheat
horizontal section. Tubes number 2 and 3 from the West Side wall should be
replaced.
Visually inspect the top of the horizontal superheat section. Check for any tubes that have
broken loose from the hanger tubes and check the bends on each end for any bend that may
be displaced and rubbing on an adjacent tube row. Also check the condition of the refractory
and flyash screens located at the rear bends of the horizontal superheat. Inspect the rear
bends for any indication of flyash erosion.
Inspect the area between the two horizontal superheat sections. Check for any tubes that
have broken loose from the hanger tubes. Visually inspect the spacer straps located on the
front hanger tubes. Check for any broken or missing U clips and to ensure that the straps are
still raised up off of the horizontal tubes in the center half of the boiler. The spacer straps
were raised because they were deflecting the sootblower steam down against the horizontal
tubes.
Special attention should be given to inspecting the sootblower lanes for retracts 10 and 39.
This area has been a very high wear area.
A. Check the condition of the tube shields located on the lower side of the upper
horizontal section. The condition of these tubes on the lower side of the upper
horizontal section should be checked at least 5 tubes deep up into the bundle.
B. Check the condition of the deflectors installed on the hanger tubes. The horizontal
tubes should be checked for sootblower erosion where they are attached to the
hanger tube.
C. Visually inspect the condition of the pad welding on the top side of the lower
horizontal section. A UT inspection of these tubes should be performed on any
tubes that are not covered with tube shields.
Visually inspect the area on the topside of the economizer. Check for any broken economizer
hangers or bent tubes. Perform an UT inspection on the topside of the bends located at the
superheat horizontal inlet header. These tubes have shown signs of flyash erosion on the
outer radius of the bends.
Inspect the area located between the economizer sections. Check for any broken hangers or
bent tubes. Check the condition of the offset bends located along the superheat backpass
front inlet header located along the front wall. These offset bends have been subjected to
some flyash erosion.
Check the condition of the refractory dam at the furnace screen tubes. Check for any broken
or missing pieces.
When performing pendant inspections special attention should be given to the first five
pendants from each side wall. Historically the first five pendants have suffered more severe
sootblower erosion.
2-17

5. External Boiler Team Member

Will perform inspections and coordinate the repairs external to the boiler. The following
inspections will be performed.
5.1 Prior to the outage (within 2 weeks of the start of the outage)
Conduct an external walk down of the boiler from 21st to 2 1/2 floor.
The items to check shall include:
Condition of the buckstays. Checking for any binding or bent buckstays. Look for any
missing or loose pins in the connector links for the boiler trusses.
Check for any external steam leaks at the sootblowers, sootblower piping and boiler vent and
drain valves.
Note any lagging or flashing that may require repair.

Check the Maximo system for workorders for any boiler vent and drain valves that
may require a code repair. Review all boiler valves that are going to be replaced to
determine if the repairs should be code repairs.
A visual inspection should be performed in the front and rear lower dead air spaces. The
following items should be checked.
A. Inspect all of the hanger rods checking for broken or missing pins. Also check for
cracking on the hanger rod lugs.
B. Check for any missing or cracked tube clips on the horizontal buckstays.
C. Perform a dye penetrant inspection where the side wall capture angle irons are
welded to the back side of the slope tubes.
D. Visually inspect the scallop bars located at the bottom of the side walls. Check
for any cracking where the scallop bar was repaired and for any cracking were the
scallop bar is welded to the tube.
E. Check for any cracking in the tube membrane located towards the inside of the
side wall scallop bars.
F. In the rear dead air space, check the condition of the centerwall tube nipple that
was cut off and plugged.
G. Check for any bent or cracked buckstays and structural steel.
H. Visually inspect for any tube membrane cracking at the throat bends and where
the slope is welded to the side walls.
2-18

Visually inspect both upper dead air spaces. Check for any membrane or skin casing leaks
and check the condition of the structural steel.
After the boiler has cooled, walk down the entire external boiler checking for any bent or
warped buckstays.
Inspect the upper boiler drum, check the following items.

A. The turbo separators, checking for alignment, loose hold down clamps and the
condition of the chevrons.
B. Missing or loose screens at the downcommer nozzles.
C. Loose or missing feed pipe hold down clamps.
D. The condition of the secondary separators.
E. Plugged lines for the boiler level devices.
Inspect the lower boiler drums, check the following items.

A. Check for any missing orifices or orifice clamps.
B. Check the condition of the screens. Look for any loose or missing screen bolts.
C. Visually inspect the drum crossover links.
Remove the hand hole covers on the superheat desuperheaters and use the borescope to check
the condition of the liner, set screws and the nozzle.
Visually inspect all of the boiler access and inspection doors recording any that require
refractory or other repairs.
Perform an internal inspection of the penthouse checking for the following items.
A. Check the condition of all the hanger rods. Check for broken rods or missing
pins.
B. Look for any tubes that exhibit signs of overheating. The overheated tubes will
have a dark discoloration and may have some scale exfoliation.
C. Look for any cracks and flyash leaks in the skin casing. Also check for any flyash
leaks at the crown seals. Special attention should be given the skin casing at the
center of the boiler. The boiler has a history of skin casing cracks along the
center of the boiler
D. Check the condition of the insulation at the superheater outlet header and links.
E. Visually inspect all of the trapeze hangers located on the connector links.
2-19

F. Visually examine all of the headers. Look for any signs of cracking at the nipples
or ligaments.(Suggest you use some NDE method than just looking dye
penetrant)
Perform and inspection on top of the penthouse. Check the condition of the upper hanger rod
attachment looking for any missing pins or cracked lugs.
2-20
3
PHASE 1 AT DETROIT EDISONS ST CLAIR PLANT
UNIT 7 AMERICAN ELECTRIC POWERS BIG SANDY
PLANT UNIT 2
Executive Summary
In an effort to improve boiler reliability at the Detroit Edison St. Clair Plant, EPRI was
contracted to identify technical and programmatic opportunities for improving Unit 7 boiler
reliability. The scope of work was limited to a portion of Phase 1 of EPRIs Boiler Reliability
Optimization project, namely boiler tubes and headers. The study was requested by the then
Production Manager Phil Thigpen from St Clair Plant. A team from EPRI spent two weeks on
site reviewing plant record, boiler inspection plans, procedures etc and conducted interviews
with key operations, maintenance and engineering staff. A list of those interviewed can be found
in appendix 1.
The report focuses on two areas - the programmatic and data collection and tube failures and
their causes and highlights a number of opportunities for improvement with respect to managing
the short and long term health of the boiler.
From the review of the data/information obtained and the interviews revealed that the current
maintenance strategy, of a periodic every three years, is in line with other utilities however the
duration is out of step with best practice in the industry. Utilities with comparable units have 4
to 5 week outages with a Forced Outage Rate of between 4 and 6 percent. . Unit # 7 has a
comprehensive boiler tube inspection plan. Tube leaks, their causes and action plans to prevent
recurrence are recorded and well managed as evidenced by the reduction in the number of tube
leaks over the past five years. The inspection plan is reactive as it only includes known boiler
tube problem areas. Inspection of the headers is limited to the superheater and reheater outlet
header. This is of concern as there is little to no information is available about the condition of
the remaining headers within the boiler.
The recent down sizing and re-organization has resulted in some uncertainty regarding the roles
and responsibilities of the newly formed service groups. Key staff have been moved to perform
other project activities, thus losing the advantage gained from past inspections. Organizational
measures and measurements were also not well understood.
To improve and sustain a high boiler reliability/availability and to minimize the future risk of
down time and subsequent production loss it is recommended that Detroit Edison St Clair Plant
3-1

Boiler Reliability Optimization Project Phase 1 at Detroit Edisons St Clair Plant Unit 7 American Electric
Powers Big Sandy Plant Unit 2
Invest into a comprehensive Boiler Reliability Optimization program to develop a detailed

Boiler Failure Defense Plan. This plan would assist in the development of a specific Boiler
Test and Inspection Plan that would cover all the major components within the boiler
envelop. This plan can then be integrated into an expanded 15 years plus Long term
Reliability Plan. This can be accomplished through EPRIs Boiler Reliability Optimization
Program
Review the basis of their current outage management philosophy and approach. The current
nine week outage on unit # 7 is out of step with other USA and international utilities even
with the current turbine problems being experienced. Huge operating and maintenance cost
saving and increased revenue can be made by reducing outage times by bring them inline
with best practice.
Implement the recommendations identified in the various section of this report.
Participating in Phase II of EPRIs Boiler Reliability Optimization Program would not only
enhance the efforts being put into the program but will give Senior Management the assurance
that the goals being set will be met. Membership details and costs related to a Phase II can be
obtained from the author.
Armed with the information in this report and available resources, St Clair plant can become the
Best Practice plant within the Detriot Edison organization for which it has been striving to
become. Given time, management commitment and coaching in EPRIs Boiler Reliability
Optimization Program, the plant team has the vision and capability, as evidence in the Boiler
Tube Inspection Reports, to successfully lead the plant into full implementation of a Boiler
Reliability Optimization Project.
3-2

3.1 Introduction
Detriot Edison St Clair Unit 7 had a history of tube failures causing considerable down time with
corresponding loss of production. Phil Thigpen, the St. Clair Production Manager, requested that
EPRI be contracted to review their boiler technical plans and identify opportunities for
improvement. The scope of work was limited to the high pressure, high temperature boiler
components tubes and headers.
Four EPRI team members - Dave McGhee, Pat Abbott, Jack MacElroy, and Jim Cossey spent
varying amounts of time on site studying documentation and conducting interviews. Interviews
were conducted with both Plant and Support staff responsible for activities on the boiler. These
interviews were conducted during the weeks of May 15 and August 29. 2000. The names of
those interviewed are listed in the appendix 1. All relevant and available data and information
on Boiler 7 was reviewed. In addition, data and information relating to the overall management
of the boiler, from an operating, maintenance and engineering point of view was reviewed. This
enable the team to gain an insight into the overall approaches taken to manage the technical
aspects of the boiler.
3.2 Findings and Observations

Based on the scope of work, the following represents the findings and observations in the two
areas, viz. programmatic/data sources, and failures and root causes.
3.2.1 Programmatic and Data Sources Assessment
A maintenance process assessment was conducted as part of the work performed in this project.
The purpose is to understand how the management processes and technologies used in the plant
affect or influence boiler operation and maintenance and therefore the long-term health. Data
was collected and evaluated and benchmarked against known best practices. The assessment
team spent a week on site reviewing this data and interviewing key site and support staff. From
the data analyzed, together with the information obtained from the interviews a spider chart
was drawn.
3-3

Score
Score
Maintenance Strategy
8
Utilization
Work Identification
6
Skills/Experience Profile
Technical Resource
Development
2
0
Metrics
Communication
Work Control
Work Closeout
Documentation
Project Management
W orld Cla ss
Change Control
Roles & Responsibilities
Operating Practices
Performance Monitoring
Boiler Inspection Plan

Technical & Action Plans
Figure 3-1
Spider Chart
3-4

Attribute
Key Findings
Boiler Maintenance Strategy
Conscience change from 2 to 3 years

Inspection and repair plan is aligned with outage
frequency
Work Process
50% of the backlog are preventive maintenance

activities.
as found and as left conditions were not being
documented
Project/Outage Management
Capital projects are detailed and well planned

Performance/Reliability of a unit returning from a
periodic outage does not meet expectations i.e. not
optimal
Unreliability and unavailability occurrences are not
investigated to determine their cause and root cause.
Operating Practices
Critical boiler metal temperatures are not monitored or

trended
Metal temperature excursions are not analyzed or
recorded
No objective evidence to indicate that boiler start-up or
shut down procedures are followed
Sootblowing procedure and criteria are not being
adhered to.
Boiler Test and Inspection Plan
Well documented boiler tube inspection and repair plan

No detailed condition assessment of all boiler headers
only superheater and reheater outlet are inspected
Metrics and targets
Overall plant performance targets exists

Boiler EFOR is not measured
Boiler tube leak target is place
Roles and responsibilities
Many of the staff interviewed were unsure of their roles

and responsibilities and that of the support group. This
was illustrated by the performance of unit 4 after its
periodic outage
Performance Monitoring
Superheater metal temperature excursions are not

investigated or recorded
No objective evidence that excess oxygen is controlled
or monitored
Skills/experience profile
Experienced people are being lost due down sizing and

retirements. This affects the transfer of technical know
how to new incumbents - operators
3-5

3.2.1.1 Maintenance Strategy

The starting point for developing a maintenance strategy is to first determine a maintenance
objective. An objective should be quantified in terms of availability and performance. These
requirements should be determined jointly between the Maintenance and Operating and or the
Production Departments. Maintenance Management is then in a position to determine the best
mix of maintenance techniques to use, resources requirements and the costs thereof.
A Maintenance strategy is then developed from these agreed objectives. It defines the process for
identifying and achieving sound operational objectives, selecting the maintenance approaches,
monitoring performance and providing effective management control. The strategy therefore
consists of a management and technical strategies.
The Management strategy defines how the business management skills are used to integrate
people, policies, equipment and practices to identify improvement opportunities.
The Technical strategy states the what, who and how the technical knowledge and experience are
used to identify and implement the best proactive maintenance, repair, service and replacement
of all equipment in line with the Plants business performance objectives. The technical strategy
therefore consists of a combination of corrective, preventive, predictive and proactive
maintenance activities where:
Corrective Fix it when it breaks
Preventive Fix it before it breaks
Predictive Fix it when it starts to break
Proactive Fix it before it starts to fail
The maintenance strategy is built primarily from the preventive and predictive maintenance basis
that exists at the plant.
3.2.1.1.1 Findings
The Preventive Maintenance (PM) base has not been updated for some years. Preventive
maintenance is not being done judged on the age and extent of PM in the backlog.
Random outage rate (ROR) defined as the sum of all forced outages and de-rates and all
unforeseen outages and de-rates divided by the available generation excluding periodic
outages and de-rates and expressed as a percentage is the availability/reliability measure
used.
Nearly all the staff interviewed could not define ROR nor did they know what the target was
for St Clair.
The planned outage frequency has been increased from two years to three years. The next
periodic is scheduled for Spring 2001
3-6

During the 1998 periodic an extensive inspection and repairs were carried out on the boiler
tubes. Since the return to service to August 2000, only two boiler tube leaks outages have
occurred. Thus meeting managements expectations/target, of one tube leak per unit per
year. This achievement illustrates the effort that has been put in by the boiler engineer to
reduce the risk of tube leaks.
Boiler tube inspections are detailed and thoroughly done, the data and information obtained
from these inspections and subsequent repair work is stored on the Boiler Engineers
computer.
Ownership of the boiler preventive and predictive maintenance basis is unclear.
EPRIs Boiler Maintenance Workstation (BMW) has been purchased however

implementation has been restricted because of the lack of allocated financial and human
resources.
3.2.1.1.2 Recommendations
Clearly define and allocate the roles and responsibilities of the various plant system
engineers.
Clearly define plant performance measures and targets to all staff. These should be tracked,
trended and displayed where everyone can see them.
Review and update the PM basis on the boiler and its auxiliaries. This can be achieved
through the use of a reliability centered maintenance package. The aim is to develop an
optimized set of PM tasks that will ensure the plant will achieve its reliability targets.
Continue with the detailed boiler tube inspection program and develop a common database,
accessible to other key people, that enables trending of the data and information and
predictions to be made.
Allocate resources to complete the implementation of the BMW throughout all Detroit
Edison plants
3.2.1.2 Work Process

A typical work process consists of four separate, but equally important steps.
Work Identification - Work is identified as corrective, preventive, predictive, or pro-active.
Work Control - Establishing a balance between the four types of identified maintenance
which maximizes the use of manpower while ensuring that the proper maintenance is
performed on equipment at the appropriate time.
Work Execution - Work is planned and scheduled. Parts and tools are staged and the proper
level of manpower is dedicated to the job.
Work Closeout Recording of as found and as left information on all work orders. This
information is maintained in an electronic database and used to plan future maintenance
activities.
3-7

3.2.1.2.1 Findings
Periodic outage duration is between 7 and 9 weeks. Boiler inspection is an on going activity
that can take up to three weeks to complete while boiler cleaning and scaffolding is
accomplished during the first week of the outage..
Boiler tube repairs made during planned periodic outages are well documented albeit in an
independent system to the CMMS. The inspection report and repair records are well written,
detailed, and complete. The repair record identifies the location of all repairs and the type of
repair made on the tubes, as well as the suspected failure mechanism.
All boiler tube repairs during a periodic are prioritized using three levels, #1 being the
highest priority.
Numerous priority # 2 and # 3 are not completed because of lack of time and resources.
Quality control plans for boiler tube repairs are used and checked by the boiler engineer.
Recommendations made by the Boiler Engineer as a result of a periodic outage inspection are
not acted on until a few months before the next periodic outage. With the current outage
frequency, this could be a three-year gap. The responsibility for acting on these
recommendations is not clearly understood or defined.
There is no backlog of planned boiler work preventive, predictive etc to be worked during
forced outages, nor is there a team dedicated to boiler inspections during forced outages.
Boiler performance data does not play a role in planning boiler outage work. The large
amount of data generated by the DCS system or stored in the PI system to assist in directing
or planning boiler work is not being used. Parameters such as tube metal temperatures,
temperature changes across reheat and superheat sections, furnace gas temperatures, heat
rate, and changes in gas pressure and flows can be used to direct inspection and repair
activities.
A long-term Reliability Management plan has been developed. The plan establishes five year
ROR, LOP, and generation targets for the St. Clair Power Plant. The plan details work to be
done at the next periodic outage, and there are no specific plans for the high pressure sections
which are approaching or exceeded the expected 100,000 hours life expectancy.
Although net heat rate is mentioned in the long-term Reliability Plan, there are no goals
established for heat rate improvement, or heat rate improvement projects listed
Conduct a technical review of outage management. This review should be and based on
sound project management principles and include a technical review of the scope of work
from each discipline. Many USA and International utilities are achieving 4 to 5 week
outages every three years. These plants are similar to St Clair Unit # 7 in size and capacity
burning Powder River Basin Coal. Shortening a 9 week outage by two weeks would have
huge impacts on outage costs and revenue.
The boiler inspection plan should be integrated into an all-inclusive Predictive Maintenance
(PdM) plan. A PdM program consists of a series of activities that are performed prior to
breakdown in order to forestall the immediate occurrence of impending failures. The present
3-8

boiler inspection program focuses on identifying and correcting existing problems.

Application of PdM methods for boiler inspections and inclusion of findings into an
integrated plant PdM database would enhance the scheduling and planning of boiler
maintenance work and contribute to reduced ROR goals. The inspection report and repair
records prepared by the boiler engineer are key indicators of the boilers condition. These
indicators should lay the groundwork of both long- and short-term planning for the boiler
maintenance program and improvement projects.
Responsibility for compiling work orders, and following through on recommendations made
as a result of the boiler inspections, should be assigned and acted on in a timely manner. The
reports should be made available in electronic form to all Detroit Edison employees involved
with the boiler maintenance program.
Advantage should be taken of forced outages to inspect available areas of the boiler.
Trigger ready forced outage plans - work orders/work packages need to be developed for
inspection of identified problem areas in the boiler immediately following both periodic and
forced outages. Thus, in the event of a forced outage advantage can be taken to perform
boiler inspection and or repair work.
The present long-term plan should be expanded to a minimum of ten years and a maximum
of twenty years. Mile stone years should be based on the outage frequency i.e. the plan
should have significant detail for years one, three, six etc. The steps to consider in
formulating this long-term plan are:
History
1. Design review
2. Construction review
3. Operating and maintenance history review
4. Inspection reports
5. Industry experiences
Evaluation of Present Condition

1. Metallurgical evaluations
2. Visual inspections
3. Non-destructive examination (NDE) prioritization
4. Destructive examination prioritization
5. Operating conditions and practices
6. Maintenance strategy
7. Database compilation
Plan Formulation
1. Remaining life assessment
2. Long-term inspection plan
3. Risk assessment
3-9

4. Short and long-term recommendations
Plan Re-evaluations
1. Review and updating of inspection plan
2. Reviews of new NDE technology
3. Inspection planning and scheduling
4. Risk assessment re-evaluation
5. Confirmation of remaining life assessment
6. Updating of plan database
Boiler performance and process data (metal temperatures, excess oxygen, steam and water
temperatures and pressures and air and gas temperatures) is available via the DCS and PI
system and should be tracked and trended. Deviations from specified should be investigated
analyzed and action taken to prevent recurrence.
Enforce the use of the CMMS historian. No Work Order should be closed out until the
craftsman who has done the work fills out this section of the Work Order. Every Work Order
should be reviewed for repeat failures, and a proper action plan developed. This is a key
factor in the continuous improvement cycle.
3.2.1.3 Predictive Maintenance Program and Technology

An optimization boiler maintenance program is made up of people, management systems,
component owners, technology specialists and predictive maintenance tools or equipment. The
key predictive maintenance tools used in a Boiler Reliability Optimization project are infrared
thermographs, vibration, oil analysis, ultra-sonic etc. Equipment owners and technology
specialists collect and collate data on various boiler components. This data is organized and
presented in an Equipment and Condition Indicator (E&CI) Matrix. The matrix specifies the
predictive maintenance technology to be used, the equipment to be monitored and when the data
should be collected, threshold/action limits/levels and actions to be taken. An example of such
E&CI matrix can be found in Appendix 2
The how, the what, and the when of condition data analysis is based on exceeding preestablished threshold levels and this information is presented in an Event Report and in an
Asset Condition Status Report. Direct causes of deviations from threshold are determined and
corrective actions are communicated to the relevant people. Threshold levels are based on
industry, design criteria and baseline data.
3.2.1.3.1 Findings
During periodic outage, visual inspections and ultra sonic measurement methods are used
extensively by the boiler engineer. The focus is on the boiler tubes.
Infrared thermography is used on electrical equipment, the boiler casing, and identification of
leaking valves.
Metallurgical analysis of failed boiler tubes is performed on a limited basis.
3-10

An oil analysis program on the fans and pulverizes is in place.
The unit water chemistry program is well controlled and the units operated within EPRI
guidelines. The unit has not been chemically cleaned for nine years, and there have been no
boiler tube failures due to waterside problems.
Report 97E08-81, dated 10/20/97, gives the results of deposit analysis on three Unit 7
waterwall tubes. The north wall sample had the highest grams/ft2 and 19.15% copper, as Cu.
2
The two other tube samples showed significantly less grams/ft and copper values
Boiler efficiency/performance tests are no longer performed.
The boiler inspection report, tube leak reports, DCS data, and PdM data need to be integrated
into a single database. The key to utilizing the condition-based information in an effective
manner is the ability to quickly access the information and to trend the data over time to
enable a prediction of when work should be performed. EPRIs Boiler Maintenance
Workstation (BMW) is one such system, which can be used to track and trend boiler tube
failure data and information. Another useful tool is the WEB based Plant view software.
Remove additional tube samples from areas where high copper had been detected (see Report
97E08-81 dated 10/20/97) and check and confirm the presence of copper.
Use the data collected by the DCS and PI proactively eg trend and track metal temperatures,
excess oxygen etc, in order to identify incipient failures.
3.2.1.4 People
Human beings are an important asset. Their reliability making and keeping commitments
has a huge impact on plant reliability. The design, manufacture, installation, operation and
maintenance of all equipment is in some way or other related to the performance of human
beings. Therefore, if human beings are reliable your equipment will be reliable.
3.2.1.4.1 Findings
The staff at St. Clair has recently undergone reorganization, with a reduction in staff
numbers. The present organization is highly matrixed, and the roles and responsibilities of
individuals are not clearly understood. Communication within the matrix is poor, with many
of the plant staff reporting they are out of the loop regarding decisions on the boiler. As an
example, there is no clear understanding, of who is responsible for following up on
recommendations made by the boiler engineer in the boiler inspection report. Is it the boiler
engineer or maintenance supervisor? At the time of writing this report, the current boiler
engineer had been assigned other duties which would take him away from doing thorough
inspections.
There is no succession planning for plant staff. Technical expertise and experience is
concentrated among a small number of staff within AMO and ESO. Plant staff are therefore
dependent on these organizations for support and boiler expertise.
3-11

An extensive on-job-training program for new operators has recently been put in place.
A Boiler Tube Failure Reduction team consisting of plant, ESO, and AMO staff should be
formed. Every tube failure should be investigated and an in-depth report issued. This report
should include a Root Cause Analysis, short and long-term recommendations, and a copy
of Work Orders written by the team. Semi-annual progress reviews should be issued giving
the status of action being taken on recommendations.
Management needs to clearly define and communicate roles and responsibilities of the Plant,
AMO, and ESO staff, and the relationship of individuals within these groups.
With the current age profile of the plant work force, it is essential to develop and put in place
a succession plan that will provide qualified people.
3.2.2 Failures and Root Causes

The section deals with the causes of failures, actions taken and the actions to be taken to mitigate
future failure.
3.2.2.1 Boiler Tubes
From 1995 to April 2000, on Unit 7 there have been 21 forced outages due to boiler tube failures.
Analysis of these reports shows 54 tube failures, involving eight failure mechanisms. See Figure
3-2 below.
The documenting of tube leaks is done via the Short Form report. The report also includes
details of the other repairs due to the consequence of the first failure.
3-12

16
14
12
A tta c h m e n ts
C li n k e r D a m a g e
F a ti g u e C ra c k i n g
10
O xi d a ti o n o f R H T u b e s
Q u a li ty C o n tro l/W e ld s
S la g E ro s i o n
S o o t B lo w e r E ro s i o n
6
W a te r L a n c e Q u e n c h i n g
1995
1996
A tta c h m e n ts
1
C li n k e r D a m a g e
1997
1998
1999
F a ti g u e
C ra c k i n g
O xi d a ti o n o f R H
T ub e s
Q u a li ty
C o n tro l/W e ld s
2000
15
3
S la g E ro s i o n
S o o t B lo w e r
E ro s i o n
W a te r L a n c e
Q ue nc hing
1
4
1
1
Figure 3-2 Tube Leak Failures 1995 - 2000
Analysis of Figure 3-2 reveals the following trends:
Failures due to water lance quenching peaked in 1995 and 1997, with four tube failures in
each year. Since replacement of the water lances, there has only been one incident, which
was due to damage prior to replacement. This pro-active approach has been successful in
eliminating this problem.
Failures due to soot blower erosion peaked in 1995 with four failures, and only one reported
in 1998 and 1999. This is due to extensive pad welding done during the 1998 outage. This
3-13

approach seems to have solved this problem for the short-term, but the large number of pad
welds threatens the units future reliability.
Failures due to clinker damage were the highest in 1997 with four failures, followed by two
reports in 1998. Installation of reinforced wear bars during the 1998 outage to date has
eliminated this mechanism.
Failures caused by the lack of adequate welding quality control has increased. Of the eight
tube failures reported in 1999 and 2000, five, or 63%, were due to poor workmanship or
using the wrong material. There were also three incidents in 1997. These failures peak
following periodic outages.
Analysis of a failed reheat tubes in 1997, showed the cause of the failure to be excessive
external oxidation. Inspections performed during the 1998 periodic outage revealed
additional fifteen tubes with excessive wall thinning due to excessive oxidation. Although
there was only one forced outage due to this mechanism, it emphasizes the importance of
monitoring, trending, and controlling tube metal temperatures.
3.2.2.1.1 Failure Mechanisms, Actions taken and Recommendations
The 1998 Boiler Inspection and Action Plan was reviewed and the following table represents the
Findings and Recommendations. The bold text represents the additional recommendations as a
result of the review process.
3-14

Table 3-1
Boiler tube Failures, Causes and Recommended Action
Attachment Failures
Findings
Cause/effect
Action taken/Recommendations
In 1997 two seal trough

failures occurred
The cause of the stress

cracks was not
determined.
During 1998 the boiler seal system was

replaced with an upgraded design to
minimize the design.
In February 2001 an
inspection revealed
several rigging/support
lugs welded onto the
waterwall and back
pass/economizer.
In many instances these

lugs if not carefully
removed can cause
future tube leaks
Welding lugs on pressure parts

waterwall, economizer etc is not a
good practice and should be
discouraged.
Burned or eroded
superheater alignment
clips are rlaced and or
repaired.
Continual welding as in
repair/replace can result
in damage to the
external and internal
tube surface of the tube,
resulting in a tube leak
Hancuffs should be used after

carefully removing broken or damaged
clips.
In September 1998 and April

1999 two failures occurred in
the horizontal portion of the
steam cooled spacer tube
attachments. This failure
mechanism was first noticed
in 1994 and 1996 periodic
outages
The cause, as recorded in

the Short Form Report is
erosion which lead to
thinning of the material
and increased mechanical
stress.
During the 2001 periodic outage

replace the water-cooled spacers
utilizing the new PIB-74 attachment
system as designed by ABB-CE
In future, when an attachment failure
occurs - tube samples should be
removed to check for possible
corrosion fatigue damage.
Clinker Damage/Slag erosion

Findings
Cause/effect
Six failures have occurred,

one in 1996, four in 1997
and two in 1998.
Localized slagging three

to four feet from each end
has been the major cause.
Investigate the cause of this localized

slagging.
During 1998 periodic outage, all tubes on
the lower slope have been replaced with
a wear barring tube extra material on
top of the tube.
Continue to inspect these tube as and
when opportunities arise - during
forced and planned outages, to
determine a wear rate from which
predictions can be made
3-15

Fatigue/Stress Cracking
Findings
Cause/effect
Nine fatigue/stress
failures have
occurred. Four in
both 1995 and 1997
and one in 1999.
These failures were all caused

water quenching from
defective/deficient water lances.
These water lances have been

replaced with an upgraded design
In February 1997 a
superheater rear wall
hanger tube
Loose/broken hanger resulting in

the tube vibrating loose and crack
caused this failure.
Compile a hangar inspection PM with all

the necessary data/ information to be
recorded so it could easily be issued
during a forced outage. This same PM
would also be issued during a periodic
outage.
Long Term Overheating/Creep
3-16
Findings
Cause/effect
In 1997 and 1998

there were one and
fifteen failure on the
reheater respectively.
Analysis of tube
samples by ESO
(Report # 99V0080001) indicated tube
failures were due to
thinning by oxidation.
The protective oxide
coating had spalled
off because of
mechanical stress
causing thinning,
Two causes were given high

temperature and mechanical
stress because of hangar support
failures. A hanger support pin
was missing.
The action taken during the 1998

periodic outage consisted of repairing
the hangers and insert 60 Dutchman
The ESO report

failed to identify or
mention if any short
or long term
overheating had
taken place. There
was no mention of
the metallurgical
aspects/condition of
the tube. The report
states thermocouple
data does higher
temperature in the
center however no
conclusions or
recommendations
are made regarding
the high
temperatures.
The ESO report therefore did not

provide adequate
information/guidance on long term
heath/life aspects of the reheater
tubing.
Regarding recommendation for hanger

inspection see recommendation under
fatigue/stress cracking.
Use UT to measure oxide thickness
across the superheater, correlate with
temperature profile thermocouples and
remove tube samples metallurgical
analysis and conformation of oxide
thickness measurement
All analysis reports of tube samples

should give recommendation regarding
future inspection strategies and or
examination requirements.
Remove during the Spring 2001 a

representative sample of tubes from
across the reheater for remaining life
assessment.

During start-up and
under normal
operation little
attention is given to
adhering to metal
temperature limits
and or recording the
extent and duration
of excursions.
Creep damage is dependent on

stress level and tube metal
temperature. Thus conditions.
Therefore conditions that deviate
from design which lead to
increase stress and temperature
will result in reduced tube life.
The importance of operating within design

parameters must be impressed upon
operating staff.
Metal temperatures should be
continuously recorded and monitored by
the operator. Rates of change and
maximum limits should be determined
and adhered to. Deviations from
expected should be recorded and
investigated. Actions to prevent
recurrence should be reviewed and
followed up to ensure compliance.
Quality Control
Findings
Cause/effect
Since 1997 a
number of pad welds
and or welding have
failed
An inspection and or investigation

revealed that the welders failed to
comply with the welding norms
and standards. Quality control of
these welds was also inadequate.
Pad welding should be discouraged

not only because it destroys the
evidence but it also can create the
potential for other mechanisms to
occur hydrogen damage. However if
it is used, criteria should given to
when and when not to use pad
welding.
Welding procedures should be
compile with and all welding should be
inspected quality control
Soot Blower Erosion

Findings
Cause/effect
Since 1995 several

tube failures have
occurred because of
soot-blower erosion.
Forty nine tube were
repaired during the
1998 periodic outage
Soot-blower erosion is
controllable. The causes are
related to excessive use
operational and improper
maintenance.
Continue with the soot blower

replacement project new helix
design.
Optimize the soot blowing sequence
and frequency by using thermal
condition in the boiler to determine
when and how long to blow.
Waterwall Fireside Corrosion

Findings
Cause/effect
During the 2001

periodic outage low
Nox burners will be
fitted
Apart from the environmental

benefits low Nox burners can
cause substoichiometric
conditions on the side and rear
wall near the burners. These
conditions nearly almost result
localized corrosion taking place.
Obtain a baseline by measuring tube

thickness local to the burners - near
the bottom burner level to about 10
feet above the top burners and the
side walls midway between the front
and rear walls at the same height of
the burners.
Ensure that the newly installed
burners are correctly optimized
measure CO and Oxygen near tube
walls if possible. High levels of CO
(>1%) and low levels of oxygen (<0.1%)
near the walls would be of concern.
3-17

3.2.2.2 Boiler Headers

Detroit Edison has a high-pressure piping and header inspection program in place for St Clair
Unit # 7. The plan reviewed did not give any details beyond the next outage. The program
includes the final superheater and reheater outlet headers and excludes all other boiler headers.
The condition of all remaining headers is unknown even though the unit has operated for well
over 100,000 hours. It is recommended that a Level II Remnant Life Assessment be undertaken.
See Appendix 3 for a typical flow chart. The findings from Level II may lead to further
examination Level III. From these assessments a Boiler Test and Inspection Plan should be
developed. The plan should cover at least 15 years. The various planning windows would be
dependant amongst other things on the particular maintenance strategy adopted outage
frequency, duration, extent of inspections etc. Table 2-2 shows the typical failure modes and
failure mechanisms found in steam drums and headers that the inspections plan should cover.
Table 3-2
Typical Header and Steam Drum Problem Areas
Header or Drum
Failure mode
Failure Mechanism
Final Superheater and

Reheater outllet
Internal Cracking
Creep-Fatigue
Swelling
High Temperature Creep
Local bulging
Creep or Thermal expansion
Reheater inlet
Sagging
Support problem and or over

temperature
Final Superheater inlet
Internal Cracking
Thermal Fatigue
All headers
External laminations
Fabrication Defects
External Cracking
Thermal Shock or expansion

Thermal Expansion Fatigue
Tube nipple Cracking

Drums
Internal surface pitting
Oxygen control in feedwater
Ligament and bore hole

cracking
Thermal and corrosion Fatigue

Thermal expansion Fatigue
External nozzle weld cracking

Economizer Inlet
Internal Cracking
Thermal and Corrosion Fatigue
Below is a brief guideline of a typical Boiler Test and Inspection Plan
3-18

3.2.2.2.1 Boiler Test and Inspection Plan Guideline
An effective inspection and test plan based on design review, historical operational data and
selective physical examination allows accurate assessment of plant condition and future life
cycle management.
Inspection of critical items of plant on design reviews together with component reviews of
results and historical data will enable effective planning of repairs/ replacement programs to be
formulated with accurate budgets. His will enhance plant availability and safe operation.
Purpose
To provide guidance in the preparation of inspection programs aimed at providing a

comprehensive plant condition database, allowing accurate assessment of current plant condition
and predicting future replacement and refurbishment requirements.
Definitions
Major Component - This is a component with an internal diameter exceeding 4 inches. It

includes headers, attemperators, internal boiler pipework etc. Heat exchanger tubing is
excluded.
Component A pressure containment operating in the creep range or subject to fatigue

together with its weldments comprising the weld metal and the heat affected zones.
Sub Components Are those components when welded together form a major component.
Component Inspection Plan This is a detailed listing for each generating unit giving a
specific schedule of components (major and sub components) to be inspected, the type and
extent of the inspection.
Inspection Report Inspection reports are generated following the physical inspection
based on the component inspection plan.
Plant Condition Review This is a rigorous review of relevant inputs that will indicate the
requirements for inspection. These include operating (including temperature excursions and
major events) and maintenance history and previous condition assessment reports.
Critical Component A component that has been identified by calculation or inspection as

having a limited life and is expected to suffer the greatest degree of life exhaustion.
Calculated values should be based on actual dimensions where possible.
Scope
The scope of the plan should cover all components (major components, components and sub
components) within the boiler envelope. The intent of the plan should be that by the time the
plant has operated for 50% of the nominal design life, at least 50% of the components have been
physically inspected and their condition known and documented.
3-19

Inspection and Test Plan Development
Compile a list of uniquely identifiable Test and Damage code.

Inspection and Test Plan Codes
Test Codes
Damage Codes
1. Visual examination
A. No Damage/ defects
2. MT
B. Defects removed < 0.1 deep
3. UT
C. Defects removed < 0.2 deep
4. -----
D. -------
Using the manufacturer flow diagrams and isometric sketches compile a list and uniquely
identify each major component, components, sub components (tee pieces, spool pieces, end
caps etc) and welds (girth, nozzle, seam etc).
Using the information from 1 and 2 above, compile a master inspection plan spreadsheet.
Use the vertical axis (rows) to list the components (major, component and sub components)
on the horizontal axis (columns) list the following information.
1. Operating parameters
2. Date of last inspection
3. Frequency of test
4. What damage/indication to identify
5. Responsible person/ system owner
6. Reference drawings
7. Dimensions
8. Material specifications
9. Test procedure
10. Database reference
11. Comment/remarks
Conduct a plant condition review and identify the critical components.
Using the master plan compile a component specific inspection plan. This plan should be
linked to a CMMS work-scheduling package. This will allow the test and inspection plan to
be included into a short or periodic outage.
Conduct the inspection and record all the relevant data.
Assess the component specific data to determine the action to be taken operate at design
operating temperature or below, refurbish, repair or replace.
Document and record actions and decisions.
3-20

3.3 Conclusions
The down sizing and re-organization has had an impact on the effectiveness of various
management processes on site as staff are getting to grips with their new roles and
responsibilities. The work flow process from work order initiation to work close out has been
affected. Planning and scheduling of short and long term work has been most affected.
The current outage on unit # 7 of 9 weeks is long in comparison to other utilities. Many USA
and International utilities have outages of between 4 and 5 weeks every 3 years. It is understood
that the outage scope of work may well be totally different however these differences need to be
analyzed and improvement opportunities determined.
The periodic outage boiler tube inspection plan and outage reports are well thought out, well
organized, and well executed. An important key to achieving a three- year outage cycle is the
continued excellence of this program. This information is stored on the boiler engineers
computer even though a Boiler Maintenance Workstation exists.
The inspection and repair efforts during the last periodic have paid dividends a reduction in the
number of boiler tube failures due extensive repairs made during the last periodic however the
number of forced outages due to tube leaks has remained constant.
The current boiler inspection plan although detailed and thorough is reactive in that it focuses on
repairing the immediate problems identified during the inspection. Areas that are not
problematic now are not inspected. These areas may well result in forced outages in the future. .
There is single central boiler tube failure database that integrates inspection results, root causes,
recommendations, failure history and metallurgical analysis
Extensive pad welding is done in lieu of resolving the sootblower replacement and or
modification/maintenance issues. Pad welding not only destroys evidence of a failure
mechanism but can be the source of a potential failure.
The Fossil Generation Long Term Reliability Plan for St Clair only includes work to be
undertaken up to the next outage. This is considered to be too short time frame to obtain the real
benefits of long term planning.
3.4 Recommendations
Consideration should be given to having a Plant Maintenance Optimization Workshop. The
workshop focuses on the broader issue surrounding maintenance. It helps identify opportunities
for improvement in all facets of maintenance, where as this report really focussed on the
maintenance issues affecting boiler maintenance. The two-day workshop is available to
members of the Plant Maintenance Optimization Target.
Compile an action plan, with allocated responsibilities and completion date, to implement the
recommendations made in the above sections of this report.
3-21

Review and benchmark the current outage management philosophy and approach with the intent
to reduce the overall outage time frame in line with best practice.
Management needs to clearly communicate the roles and responsibilities of the newly formed
service groups to all personnel involved. Expectations need to be clearly defined and individuals
held accountable for their successes and failures.
Compile a root cause analysis process guideline to be complied with by all Detroit Edison plants.
Train all staff in the basic concepts of root cause analysis and a number of engineers at each site
on the details of applying root cause analysis to incident and problem resolution.
Assemble a Boiler Tube Failure Reduction team to investigate ALL boiler tube failures. Their
responsibilities would be, determine root causes, recommend solutions, follow up on
recommendations, issue annual reports on the boilers long term health, maintain an integrated
database, and, put together and continuously update the long term plan for the boiler.
Complete the soot blower upgrade project, assign a higher priority to soot blower and water
lance PMs, and analyze unit operating data to optimize the combustion process and decrease soot
blowing frequency. These actions would greatly decrease the number of pad welds needed to
counteract soot blower damage.
3-22

Appendix 1: List of interviewees

Melanie McCoy
Plant Manager
Phil Thigpen
Reliability Engineer
Bob Sausser
Supervisor Plant Operations
Dan Kuhlman
Supervisor Plant Operations
Dan Lavere
Predictive Maintenance Coordinator
Tom Seaton
General Foreman
Mike Mallon
Boiler Reliability Engineer
John Shinske
Unit #7 Foreman
John Renehan
Unit # 7 Planner
Harry Mueler
Unit # 7 Engineer
Bill Fielhauer
Unit # 7 Engineer
Mike Bennet
Water Chemistry Engineer
Bill Harvey
Instrument Shop Foreman
Willie Myers
Water Laboratory Technician
Fred Cutter
Operations Coordinator
Dave Hurt
Shift Supervisor
Jack Miketich
Shift Supervisor
Bill Hornby
Shift Supervisor
Barry Wagner
Supervising Operator
Jeff Hat
Charles Burmann
Jim Jordan
Ken Brinker
Dave Stephan
Training
3-23

Appendix 2: Example of a Typical Equipment Condition and Technology

Matrix
3-24

Appendix 3: Remnant Life Assessment Flow Chart for Thick Section
Level II Assessment: Thick Section Parts
Assemble design and service information

Design: * Dimensions * Pressure
* Materials * Temperature
* Minimum
* Stresses
creep-rupture
Conduct careful visual inspection. Are

there cracks at ligaments or at
penetration welds?
Service: * Boiler running hours

* Past repairs and replacements
* Dimension and composition checks
Conduct detailed inspection by

best NDE methods available.
Measure dimensions of all
crack-like defects.
Yes
No
Level III Assessment
Answer key questions
Has unit significantly

exceeded design P and/or T?
Yes
Will future service involve P

and/or T above original design?
Yes
Yes
Level III Assessment: Thick

Section Parts
Attach thermocouples; determine

temperature distribution. Monitor
temperature at hot spots to obtain
representative sample.
Perform detailed stress

analysis by finite element or
finite difference methods.
Level II
Assessment
Has failure history of boiler

been excessive?
Using actual temperature and

pressure, calculate life fraction
expended and remaining life
Yes
Perform transient and steadystate thermal analyses by

finite element or finite
difference methods.
Perform fatigue, creep and

fracture mechanics
analyses. Calculate
remaining life to crack
initiation and to fracture
after crack growth.
Steam temperature records not

available?
No
Is remaining life greater than the
projected life extension?
Is remaining life greater than

the projected life extension?
Level I Assessment
No
Yes
Set inspection
interval; maintain
accurate operating
records
Using design and basic

operating data, calculate hoop
stress, life fraction expended
and remaining life
Level III Assessment
Calculate remaining life
RL = (1 LFE)tR
Is remaining life greater than the

projected life extension?
No
Make repair or
replace
decision
Estimate costs
and schedule
Yes
Set
inspecti
on
interval;
maintain
accurate
operatin
g
records
Input to plant
and corporate
plans
Yes
No
Set inspection
interval; maintain
accurate operating
records
Level II Assessment
3-25
4
PHASE 1- ROOT CAUSE ANALYSIS AT ARIZONA
PUBLIC SERVICES COMPANYS FOUR CORNERS
AND CHOLLA PLANTS
FOUR CORNERS PLANT

1999 Lost Generation Incident Review
In an increasingly competitive environment, the ability to reliably produce the lowest cost power
is essential for the long-term survival of a utility. With this vision in mind Arizona Public
Services Company (APSC) requested EPRI to review the 1999 Lost Generation Action Plans
from the Four Corners Plant. The intent of this review was to provide assurance or not that the
true root causes of lost generation have been identified, and appropriate action plans are in
place to address these causes.
The review took place between 30 May to 9 June 2000 on site at the Four Corners Plant. Ten
incidents, resulting in a loss of 728,978 Mwh were reviewed. Of the ten incidents reviewed only
one was adequately documented - backing up the plants root cause analysis efforts. The review
team, two from EPRI, one from Cholla Plant and one from Four Corners Plant, tried, through a
series of interviews, to re-construct the incident. More than twenty staff members were
interviewed.
Two methods, Behavior Justification and Event and Causal Factor Charting were used to reconstruct the incidents. Diagrams/charts were constructed from information that depended,
almost entirely, on the interviewees' long-term recollection of incidents. Verification of
information was difficult, as there was a lack of formal incident-specific documentation and
because of the time that has elapsed between the incidents and the analyses. In many incidences,
perceptions and memory of what happened and in what sequence varied, emphasizing the
importance of performing Root Cause Analysis in a timely manner, and then documenting the
results.
Based on the information obtained during the two weeks at the plant, the EPRI team uncovered
many opportunities for improvements in operations, equipment maintenance, inter-departmental
communications at all levels of management, human performance, root cause analysis
documentation and engineering response. The Lost Generation Action Plans as developed by the
4-1

Boiler Reliability Optimization Project Phase 1- Root Cause Analysis at Arizona Public Services Companys Four
Corners and Cholla Plants
plant need these improvements to ensure that they have a sound base. Documentation of incident
investigations, no matter how minor, provides an audit trail to ensure a historical record of what
has occurred at the plant, how it was analyzed, what the cause of the problem was and how it
was solved. Lack of this audit trail often leads to repeat incidents because staff are not aware of
what has happened previously/how problems were solved, often leading to short-term,
inadequate solutions, rather than structured, long-term and cost-effective solutions.
No formal structured root cause analysis process exists. Some plant employees have been
trained in a root cause analysis technique, but do not use it as it was intended to be used. The
staff members were cooperative and mostly forthcoming and interested in using a Root Cause
Analysis tool to solve long-term problems at the plant. Root Cause Analysis training/re-training
needs to be implemented plant to ensure that staff members are confident in their abilities to
solve problems that arise.
Each incident reviewed has a number of recommendations. These recommendations need to be
evaluated and implemented. The main recommendation is that a formal, structured approach to
incident investigation needs be developed and implemented at each plant. A corporate directive
should be developed to give guidance on the requirements. As a start, all unit trips/forced
outages, load loss, and failures to meet expected outputs, should be subjected to thorough root
cause analysis. Incidents where plant damage, exceeding a predetermined amount, has occurred
should also be investigated and any significant capital expenditure needs to be backed up with a
justification that is based on a thorough root cause analysis.
4.2 Introduction
In an ongoing effort to improve reliability and reduce maintenance costs in the APS system,
EPRI was contracted to review the 1999 Loss Generation Action Plans at Four Corners Plant.
The main objectives of this review were:
4.2.1 To provide assurances that the true root causes of lost generation have been identified,
and appropriate action plans are in place to address these causes:
4.2.1.1 determine and verify the true root causes of technical and/or production problems,
and
4.2.1.2 identify preventive actions to effectively manage the identified problems and
minimize the affect on availability.
The team consisted of, two from EPRI and two from APSC, one from Cholla Jack Stant and
one from Four Corners Plant Mike Rosner. The reviews took place between May 30 through
June 9 on site at Four Corners Plant. More than twenty staff members were interviewed
regarding the 10 incidents reviewed. These incidents represented a loss of 728,978-Mwh at the
Four Corners Plant. The incidents selected represented the major Mwh losses and were
prioritized by the plant management team. The selection was based their significance, available
documentation, and availability of staff having knowledge of the incidents.
4-2

The Behavioural Justification method of analysis was used to analyze some of the incidents that
occurred some APS personnel have been trained in this method. Events and Casual Factors
(E&CF) Charting was used as an alternative method. In this method the sequence of events is
charted chronologically, from left to right, and information/conditions relating to events in the
sequence are charted directly underneath the events. Events are shown as text within rectangles,
joined by arrows, ending in a circled unwanted event (reason for the analysis). The
information/conditions are shown in ovals, joined by lines that show their association. Dotted
ovals or rectangles indicate information that still needs to be verified. Causal factors are
identified by asking the question Which of these factors, if eliminated, would have either
prevented the incident from occurring or at least minimizing its impact? From these causal
factors Root Cause are determined. Root Cause is the cause/s which, if eliminated, would
prevent this event recurring or minimize its impact.
4.3 Plant Description

Four Corners Power Plant is owned by Arizona Public Service and is situated in the Four
Corners region of New Mexico. Units 1 & 2 are Riley Stoker front fired, natural circulation
boilers, rated at 170 Mw each. They were placed in service in 1962 and 1963 respectively. Unit 3
is a Foster Wheeler front fired, natural circulation boiler rated at 220 Mw and went into service
in 1964. Units 4& 5 are supercritical B&W units rated at 755 Mw each. They went into service
in 1969 and 1970 respectively. All units burn high ash Western coal.
4.4 Incidents investigated

The incidents selected and the Mwh lost are listed in the following tables:
Four Corners - Megawatt-hour loss summary report-fourth quarter 1999
Mw Hour Lost Area
Mwh. Lost
Unit 5 #4 Control Valve
197,471
Unit 4 Air Pre-heater Failure
166,744
Unit 3 Turbine Copper Fouling
100,628
Unit 4 LP Exciter Field Ground
45,229
Unit 3 Boiler Tube Leaks
39,378
Unit 4 Flyash Erosion
39,288
Unit 4 Battery Failure
38,413
Units3 Pulverizers
34,881
Unit 4 HP Exciter Field Ground
33,500
Unit 3 Scrubber, and Recycle Pumps
33,446
Total
728,978
4-3

4.4.1 Unit 5 #4 Turbine Control Valve chatter - 197,471 Mwh lost

Limited documentation was made available on this incident. To construct a sequence of events
the following people were interviewed - Jack Rogers, Willard Billey, Dave Saliba, Joe Rogers,
John Bates, and Randy Wilde
Incident Description
From June 1 - 20, 1999, and from July 1 - Dec. 27, 1999, Unit 5 was de-rated by 6 % due to # 4
turbine control valve problems. On June 17, 1999 during a tube leak outage the valve linkages
were repair and the valve operated satisfactorily for 11 days. On July 1 an operator reported that
the valve was once again chattering and later slammed shut, and could not be re-opened. The unit
was operated with the load limiter set at 94% until a boiler tube leak outage on Nov. 26. During
this outage the control valve was checked and found to be operational and the governor pilot
valve was rebuilt using re-conditioned parts and other parts, which were not from the original
equipment manufacturer (OEM). The unit was brought back on line, and continued to run with
the load limiter set at 94%. On Dec.25 the unit trip because of a switchyard failure, the unit was
returned to service the same day with no load limitations due to the # 4 control valve.
Root Cause Analysis Chart
Figure 4-1 was drawn based on interviews and operator logs. Knowledge of the problem and
actions taken, or not taken, differed between individuals. This made it difficult to accurately
record the sequence of events. However, the information as reflected in the chart was accepted
by all those interviewed as a fair reflection of what happened.
An analysis of the chart identifies a number of opportunities, that if investigated, at the time,
would have eliminated the event or at least minimized the impact of the event. These are:
There was no attempt to inspect or repair the valve during the June 11 boiler tube leak
outage.
On July 1 it was reported that the valve was chattering and then had slammed shut and could
not be reopened. This was accepted without an investigation and the unit ran de-rated until
November 26. Reason for not investigating at time could not be established.
During the November 26 outage the value was checked and verified as operational. However
the unit ran de-rated until the December 25 outage. The reason for this continued de-rating
could not be established
No valve work was performed during the December 25 outage, however after the outage
Operations were informed that the valve could be operated. The unit was returned to service
with no de-rating attributed to the control valve.
To determine the root cause of why so many Mwh were lost, the above points need to be
investigated further. This could not be accurately done because of the uncertainty surrounding
the sequence of the events from June though December 1999. However, given that the facts in
the chart are a fair reflection of what happened, two causes come to mind that if corrected would
4-4

have minimize the impact of operational failure of the valve. These relate to problem ownership
no person took responsibility to solve the problem and communication between individuals and
between departments.
An operational test of the value undertaken at the time of the review showed that the valve
continues to chatter. The cause of the chattering is still unknown and waiting for suitable outage
to determine the cause.
4.4.2 Unit 3 Scrubber unit failure - 33,446 Mwh lost
No documentation was made available on the two incidents that made up this Mwh loss,
however Phil Valencia provided some background information.
These two incidents involved the following deficiencies.
Increasing rate of failure of scrubber recycle pumps. Scrubber debris was being caught in the
pump impeller, causing pump imbalance. This imbalance resulted in pump bearing failures.
Repeated blockage of the overflow and emergency overflow pipes with rubber lining and
debris. Rubber-lined piping is used throughout the scrubber system to protect the piping
from the highly erosive fluids, which it handles. Portions of this lining break off and block
the overflow pipes.
Repeated unit outages because of high levels in the scrubber unit. Slurry flowing out of the
emergency and or overflow pipe is used as an indication of level. These overflow pipes can
get blocked from excessive amounts of debris is in the slurry system.
Structural failure the Dentist Bowl. The bowl is attached to a telescoping assembly used
to maintain bowl position. Debris collects in the assembly and prevents free movement. This
caused high stresses on the bowl when operated which result in eventual failure.
The above resulted in two trips of Unit 3.
The first incident was the result of a high/high level caused by blocked overflow pipe. The
emergency overflow was also plugged with debris. The scrubber flooded preventing flow.
The delta pressure across the scrubber increased resulting in a unit trip - high furnace
pressure
The second incident involved a recycle pump failure. Refractory had fallen off a metal skirt
in the scrubber and jammed in the recycle pump impeller causing an imbalance. This
imbalance caused the pump bearings to fail resulting in a pump tripped. This caused the unit
to trip.
4-5

Root Cause Analysis Charts

Figure 4-2 shows the correctable opportunities identified for all four deficiencies highlighted the
Behavioral Justification methodology. Clearances had been opened on the dentist bowl
reducing the risk of any jamming of the telescopic section.
Figure 4-3 represents the E&CF chart was used to analyze the lining failure and blockage
problems. Since the piping failures and scrubber level problems were interrelated, they were
analyzed together. At low temperatures the gum rubber separates from the metal surface cold
wall effect forming blisters. These blisters eroded by the flowing slurry, break open and tear
resulting in pieces of the lining coming off the pipe. These pieces of rubber block both the
normal and emergency overflow pipes. Operators use the flow through the normal overflow pipe
as an indication of correct level in the tank. Figure 43 highlights the correctable opportunities.
Using a lining, which is more resistant to cold wall effect. Experience at Cholla Plant has
proved that the use of a different rubber - chlorobutyl, which has a higher resistance to the
cold wall effect, significantly reduces blistering.
A more sophisticate method to determine the level in the tank should be installed rather than
use the normal overflow pipe.
Figure 4-4 is an analysis of scrubber recycle pump failures and highlights some correctable
opportunities.
The pumps have urethane-coated impellers in which debris can easily become lodged, causing
rotor imbalance. This imbalance causes the seals fail resulting in slurry entering the bearings,
resulting in bearing failure. The major source of debris is refractory, which had been applied to a
metal skirt in the upper portion of the scrubber in an attempt to protect the skirt. Moisture gets in
between the skirt and the refractory. The resulting corrosion causes large pieces of the refractory
to break off and fall into the sump. These pieces get sucked into the pump suction and jam in the
pump impeller.
Further investigation is required design review, to determine whether the skirt is required for
the correct operation of the system. There was some doubt by members of the review team of its
purpose. If, in deed it is required, then it should be replaced with a corrosive resistant material
rather than a material with refractory.
As and when opportunities present themselves, the urethane impellers should be replaced with
white iron impellers. These impellers are more robust and fit for purpose in the scrubber
environment. This has proved to be successfully at Cholla
In addition suitably sized suction screens need to be installed. This is an inexpensive activity if
undertaken during a suitable outage. These screens would be the first lines of defense to prevent
over sized material entering the pump.
These items are not specifically addressed in the 2000 Action Plan but can be accommodated in
the $500k is budgeted for scrubber work.
4-6

4.4.3 Unit 4 Loss of Booster Fans
38,413 Mwh.
Mel Begayle and Richard Bagienski provided some background information, as there was no
documentation available.
Regular preventive maintenance tasks performed on the twelve years old Booster fan control
backup batteries had shown that the batteries needed to be replaced.
For some unknown reason, AC power to the DC converter was interrupted; this required the
controls to be supplied by the batteries. The DC voltage dipped below 108 volts. This dip in
voltage caused the BETA alarm panel to reset, since the batteries were unable to maintain the
required voltage. The booster fans tripped, tripping Unit 4. Batteries purchased for another
purpose were used for replacement and another set of batteries ordered. The alarm panel was
also changed.
The E&CF chart - Fig. 4.3 shows the sequence of events. Lack of any documentation and
relying on the memory of the individuals involved made it difficult to re-construct the sequence
of events and hence determine the root causes. The direct cause of this incident was the poor
condition of the batteries. However the following was determined from the interviews:
electrical maintenance staff did not know how to correctly check the batteries
Incorrect meter was used to perform the battery check.
No action had been taken of the battery deterioration as noted on the preventive maintenance
(PM) schedule. The crystal growth in the batteries had been visually verified and noted on
the PM.
Additionally, a crossover switch between the Unit 4 batteries and Unit 5 batteries has been
installed. The proper meter is now being used and the PM has been rewritten. All the actions
discussed are included in the Lost Generation Action Plan.
4.4.4 Unit 3 Pulverizer Pinion & Bull Gear Failure - 34,881 MW Hrs.
Bill Burnett and Calvin Charlie provided the information to re-construct this incident.
This incident involved the following:
The foundation of 3B pulverizer had failed due to excessive vibration caused by incorrect
alignment of the bull and pinion gears. It failed at the same place where a similar repair was
made ten years previously.
4-7

The pinion gears, which had been purchased from different suppliers, did not meet the
original specification and were installed by maintenance staff.
A bull gear known to be out of specification was accepted and installed.
Failure of the pulverizer lube oil pump.

The Behavioral Justification methodology was used to analyze this incident. This is shown
graphically in Fig.4.4. Four correctable opportunities were identified.
A procedure needs to be developed with criteria on the acceptance/rejection of

equipment/components that do not meet specification/requirements. Plant management had
accepted a bull gear, which had been improperly manufactured and had installed it. Reason
for this decision could not be established.
A Quality Assurance/Quality Control (QA/QC) process needs to be developed and followed

to ensure that equipment maintained or manufactured by a supplier conforms to minimum
requirements and is fit for purpose. This is especially essential with components like pinions
and gears. Pinion gears are purchased from several suppliers and no QA or QC is performed.
Operator rounds need to be enforced. The failed lubrication oil pump went unnoticed for an
unknown length of time. The lack of lubrication caused the gears to overheat and contributed
to the failure.
The Lost Megawatt Action Plan does not address this issue. Replacement of the pulverizers is
being considered. However, the problems discussed above will not be solved with new
pulverizers. New equipment will not resolve procedural, management, or communication
problems.
4.4.5 Unit 3 Back-pass Flyash Erosion 39,378 Mwh
David Hughes, Drexel Pruitt and Duane Pilcher shared with the review team their insight and
knowledge of the erosion in the back-pass of Unit 3
Problem Description
Flyash erosion and the resulting tube failures have been the major cause of forced outages on
Unit 3. Changes in coal quality higher ash content combined with a furnace size, which was
marginal for the originally specified coal, and continued operation at loads equal to or greater
than design output have all contributed to fly ash erosion. Extensive shielding and numerous
modifications have further increased the gas/particle velocities, which has compounded the fly
erosion problem.
4-8


Fig.4.5 identifies a number of correctable opportunities. These all need to be evaluated
separately and collectively. The purpose is to reduce gas velocities. Some of them are related to
poor pulverizer maintenance. However, a significant contributor is the number of unqualified
modifications. In the past, a number of modifications have been done to solve an immediate
problem without considering the effects of such a modification on other areas of the boiler. Many
of these modifications have indeed met the original intent and created an erosion problem
elsewhere in the boiler. This is compounded by the lack of documentation on the number and
rational behind the modifications. A Backpass study scheduled for completion in Sept. 2000
is in the Lost Generation Action Plan. It is recommended that an E&CF chart be drawn/used to
analyze all past modifications made to the unit. This would be a daunting task, but extremely
useful in resolving and understanding all the erosion problems.
4.4.6 Unit 4 Air Pre-heater Failure 166,744 Mwh.
Ron Bertram, Jim Justice and Randy Wilde provided some detail about this incident, as there
was no documentation regarding this failure.
Problem Description
Two modifications to the air pre-heater had been done to facilitate the early operation of the bag
filter during unit start-up. These were to slow down the rotational speed of the pre-heater and
delay its in-service operation during unit start-up. The net effect of these modifications on the
air pre-heater was an increase in the number and extent of thermal cycles. This resulted in the
thermal fatigue failure of a cold end basket retaining frame/clip. This allowed the basket to shift
and bind the air pre-heater causing extensive damage.
Figure 4-8 diagrammatically shows the causes and correctable opportunities. The two
operational changes significantly contributed the accelerated thermal fatigue failure of the cold
end basket-retaining frame. Both of these conditions have been corrected and an extensive
inspection and repair effort is included in the Lost Megawatt Action Plan.
4.4.7 Units 4 Flyash Erosion 39,288 MW/Hrs.
Mike Rosner provided the background to this problem.
Problem Description
Units 4 & 5 have a history of boiler tube failures due to flyash erosion in the back passes. This is
because of a marginal boiler design and over firing conditions. When over firing for extra
megawatts, operations increases excess air in order to prevent clinker formation. This results in
4-9

higher gas volumes and increased velocities. The short-term effect is less clinker formation. In
the longer term this mode of operation increases flyash erosion resulting in unwanted tube leaks.
This significantly impacts the reliability of the unit
Fig. 4.7 represents diagrammatically the information provided. The flyash erosion problem is
the result of a marginal boiler design as in furnace size BTU/square foot, choice of the most
economical fuel and the operational philosophy. Erosion rate is dependent on a number of factors
velocity being a major factor. The relationship between erosion rate and velocity, is
exponential - velocity to some power between 2 and 3. So, velocity hence gas flow should be
kept at a minimum and under control. A flue gas velocity distribution test is scheduled for 2003.
Extensive efforts are planned in the Lost Megawatt action Plan to repair and mitigate the effects
of flyash erosion.
4.4.8 Unit 3 Turbine H.P. Nozzle Pluggage 100,628 Mwh.
The following provided some documentation and described the turbine pluggage problem - John
Huff, Joe Baker and Dave Viser
Problem Description
Shortly after switching from hydrazine to Eliminox for dissolved oxygen control in 1993, the
Unit 3 turbine/generator MW output started to decrease. In 1996 an investigation was launched a fault tree analysis was drawn. This analysis identified that the reduced output was due to a
restriction in the high-pressure (HP) turbine inlet nozzle. Results of a chemical cleaning the
nozzle showed the deposits to be copper. Analysis of the economizer inlet water indicated that
the copper was coming from the feedwater system. It was assumed the source of copper was the
condenser. The HP turbine was cleaned several times, and each time nine to twelve pounds of
copper were removed
The Behavioral Justification methodology was used to analyze the problem. Figure 4-10 shows
four correctable opportunities.
The replacement of hydrazine with Eliminox in 1993 followed an industry wide trend to
remove hydrazine, a known carcinogen, from power plant sites. Substitutes were not as
effective and many units observed an increase in copper corrosion and subsequent copper
carryover in units operated in excess of 2300 PSI-drum pressure. Early in 2000 the unit has
returned to hydrazine treatment. Hydrazine levels are adjusting to maintaining a negative
oxidizing/reducing potential at the economizer inlet.
The condenser is to be replaced with 29-4-C, a high performance stainless steel.
It is also planned to replace copper metallurgy feedwater heaters with stainless steel as the
need arises.
4-10

These actions will prevent further migration of copper corrosion products into the boiler
systems. However, there is copper in the water wall and superheater tubes which can be a
major contributor to tube failures. A chemical cleaning of the waterwall tubes was done in
Feb. of 2000.
4.4.9 Unit 4 LP Exciter Field Failure 45,229 Mwh.

Richard Bagienski was interviewed. After the interview details of the incident were provided.
Unit 4-exciter failed resulting in a unit trip. Inspection of the brushes and brush holders revealed
that some misalignment between brush and brush holder tension arm. The brush arm arced and
overheated the collector ring, which, in turn, caused other brushes to arc. The increase in
resistance tripped the unit on low excitation.
Root Cause Analysis
Documentation provided was clear, concise, and complete, and is included in the appendix. The
problem was quickly identified and the appropriate actions taken. Access to check brush
alignment is difficult. The importance of ensuring correct alignment was not identified on the
PM. This can be considered as the root cause. The PM has now been rewritten to include an
inspection of the tension arms. The Lost Megawatt Action Plan has extensive Exciter work
planned for both Units 4&5 in 2000.
4.4.10 Unit 4 HP Exciter Field Failure 33,500 Mwh
Richard Bagienski was interviewed.
Problem Description
Unit 4 was taken off line after analysis of an exciter field ground alarm indicated that there was a
ground in the field about at 50% through the winding. After the unit was taken off line, the
exciter field was isolated and the ground was verified. A copy of APS documentation covering
this event is included in the appendix.
Root Cause Analysis
As in the LP exciter event (see 4.9.1), the problem analysis was well done. The exciter was
shipped to General Electric for inspection and repair. The inspection revealed that a previous
rewind was substandard. In 1992 an alternate vendor was used to do the rewind. This
substandard rewind was considered to be the direct cause of this incident, Lack of proper
QA/QC is the root cause of the problem.
4-11

4.5
Findings
At each of the stations, the review team consisted of the two EPRI employees and a staff
member from the other station involved in this exercise. This was a very positive aspect of
the review as experiences and problem solutions at both stations were shared. The
opinions/ideas of peers were welcomed.
Root Cause Analysis is done by exception, i.e. as and when requested by management.
Some staff members were trained in the Root Cause Analysis technique taught by Bobby
Jones and Associates Behavioural Justification, but they dont regularly use any formal
analysis technique.
Root Cause analysis, commonly, consists of group brainstorming sessions, where solutions
(often short-term) are formulated and actions allocated.
These solutions are not documented and limited follow up is done on allocated actions.
There was very little documentation available from which the review team could gather
information. Therefore, in an attempt to reconstruct the incidents key information had to be
obtained from staff members who had knowledge thereof. E&CF charts or Behavioral
Justification diagrams were constructed from this information and causal factor determined.
Of the ten lost generation incidents reviewed, only one had some usable documentation
Unit 3 Turbine Nozzle plugage. This investigation was started in 1996.
In more than 70 % of the reviewed incidents, it was found that staff chose to rather
replace/fix/modify equipment or plant, instead of assessing root causes.
No documents were provided to support the1999 Lost Generation Action Plans and the
Action Plans, therefore, could not be validated as required in the Scope of Work for this
project. In almost all the incidents reviewed /reconstructed, the fix repair/replace or
modification did align with those recorded in the 1999 lost Generation Action Plan.
Preventive/Predictive Maintenance schedules/activities are not implemented, as staff are

focused on corrective maintenance, as in crisis management.
There is a lack of problem ownership, as responsibility for resolution of problems is not

specifically allocated, e.g. the turbine control valve chattering.
The long-term health of the plant is neglected. The plant is run in excess of the design
maximum continuous rating, e.g. boilers are over-fired to a point where clinkers start to
form; continuous sootblowing is then used to remove this build up and this causes erosion
which results in long-term damage and/or a tube leak.
Project kick-off and exit meetings were held to confirm scope of work and give feedback
respectively. Considerably less management staff attended the exit meeting.
4-12

4.6 Conclusions
The findings of the EPRI team led to the following conclusions being drawn:
4.6.1 Formally-documented Root Cause Analysis is not being done at the Four Corners. The
most common reason for not doing a root cause was, we are too busy putting out fires.
This made verification of the actions identified in the 1999 Lost Generation Action Plan
difficult, however through interviews actions could be verified.
4.6.2 Although no formal analysis is being done, the actions (solutions) identified in the 1999
Lost Generation Action Plans are considered short-term quick fix solutions.
4.6.3 Knowledge of a Root Cause Analysis process was limited even though staff had been
trained in an analysis technique.
4.7 Recommendations
4.7.1 A formal structured Root Cause Analysis process needs to be implemented. This process
should be developed based on a corporate directive requiring those major incidents
(greater than a $X, 000 value to be determined by Corporate), unit trips and a
significant unexpected maintenance expenditure be investigated to ensure repeat
failures/incidents/trips do not occur.
4.7.2 Staff should be trained/retrained in a Root Cause Analysis technique that includes Human
Performance aspects.
4.7.3 The various recommendations identified in Section 4.4 Root Cause Analysis should be
evaluated and implemented.
4.8
Appendix 1: Root Cause Analysis Charts
(Excel file)
Note:
1. A number of the charts are incomplete as there was limited documentation and the allocated
time per incident was constrained. However they do reflect the information given or
presented by the various interviewees.
2. The charts need to be read in conjunction with the text in the main body of the report
4-13

Boiler Reliability Optimization Project Phase 1- Root Cause Analysis at Arizona Public Services Companys Four Corners and Cholla Plants
6\1\99
6\1\99
6\11- 6\13
7\11- 11\26
S .S . is notified
that #4 valve is
s tuc k in clos ed
pos ition
Operating
with load
lim iter at 94%
Unit operates
with load
c urtailed
Control operator
attem pts to
open
valve.
Unit is run at overpressure to m eet

load requirem ents .
No written
verification
No investigation
initiated
Figure 4-1
Unit 5, #4 Control Valve 197,471 Mwh
4-14
Turbine planner &

Operations M anager
verify that #4 c ontrol
valve is operational.
Unit operates
with no load
c urtailm ent
No chattering
detected
B rack et
m ac hined
E videnc e of
previous
failure
No failure
analys is of
bolts
7\???
S .S . reports
#4 valve is
c hattering
#4 c ontrol valve
link age repaired
B roken
bolts
replaced
72 hour outage
required for
inspec tion
6\21
S .S . reports that
#4 valve is not
chattering
Lost
Opportunity
Load lim iter set

to 93%
7\1
6\20
Unit shut down

for tube leak
No attem pt
m ade to repair or
inspect valve
V alve clos ed to
stop chatter
Turbine
planner
notified
6\17- 6\19
Unit shutdown
for tube leak
Unit curtailed
to 710 M W 's
S .S . Reports
#4 control valve
chattering
Load lim iter

set to 94%
11\26-12\2
12/2-12/25
12\25
Unit shut down

for tube leak
repairs
Unit continues to
operate with load
lim iter at 94%
Unit shutdown
due to switchyard failure
#4 c ontrol valve
chec ked and found
to be
operational
Governor pilot valve

rebuilt with
non-OE M
rec onditioned parts
Operators unaware
valve c hec ked and
found to be
operational
Curtailm ent
lifted after
startup - Unit
at full load
197,471 M wh
lost due to #4
control valve
problem s .

Lost Mwh
due to Scrubber
O utages
Recycle pump
Rubber-lined
High level of slurry
failures
piping failure
in scrubbers
Foreign objec ts
c ause im peller
imbalanc e
P um p
c avitation
I.D.
Refac tory falling

off of s k irt
Cold wall effec t

pipe blis tering
C.O . Material
Overflow
plugged
with
debris
No level
indic ation
S truc tural
failure of
Dentis t B owl
B lowdown
not
s uffic ent
Teles c ope
jam m ed
C.O .
selection
No level
indic ation
No level
indic ation
Tight
c learanc es
C.O .
C.O .
C.O .
C.O .
Debris
between
pipe
clearanc es
N.C.
1. Replace
skirt w/o
refactory ,
Screens,
W hite iron
pumps
Figure 4-2
Unit 3 Scrubber Problems 33,446MW/Hrs.
4-15

B listering in
rubber-lined
piping.
Rubber
separates
from piping
No level
ins trum entation
Figure 4-3
Unit 3, Scrubber Problems 33,446MW/Hrs.
4-16
dP across
scrubber
increased
suddenly
*
*
S crubber debris
plugs em ergency
overflow
Pipe diam eter

is 14"
P ipe diam eter

is 14"
Norm al overflow
is used for level
c ontrol
High water
level block s
gas flow
Rubber piece
20" diam eter
Gum rubber very

sus ceptible to
c old wall effect
Gum Rubber
vers us
Chloro B utyl
Seperated
rubber plugs
normal overflow
Overflow pipe is
long and alm os t
horiz ontal
Fans were fully loaded - 100%
Overflow pipe is
long and alm os t
horiz ontal
Unit 3 trip
due to
high furnace
pressure

Refac tory
applied to
m etal s kirt
Refac tory
falls off
m etal s kirt
No s tuds or
m etal
hangers
d
Purpos e of
refac tory is
not k nown
Bas e m etal
c orrodes
Refac tory lodges

in im peller
Rotor bec om es
im balanced
No pum p
s uction
s c reens
Im peller
des ign
Urethane
coated
Free diam eter

pas s age is 5"
Pum p
fails
Slinger rings
wear out
Unit derated
Loss of M W
due to
rec yc le
pum p failures
E nvirom ental regs

require pum p to be
in s ervic e
P ac k ing
fails
Figure 4-4
Unit 3 Scrubber Recycle Pump Problems 33,446MW/Hrs.
4-17

B attery PM
perform ed
E lec tricians
vis ually inspect
batteries
Incorrect voltage
m eter used to
test batteries
DC inverter
output voltage
drops .
DC voltage
dropped to les s
than 108 V
*
Cry stal growth
in batteries
B atteries
were bad
A C power to
inverter was
interrupted
Indication of
end of life
Data not
c hec k ed or
analy z ed
*
Replac em ent
batteries were
not ordered
B ooster
fan trips
Figure 4-5
Unit 4, Loss of Booster Fans, 38, 413 MW/Hrs
4-18
Unit 4 trips
due to los s of
boos ter fans
New batteries
ordered
B E TA alarm
panel res et
*
B E TA panel
s ensitive to
input voltage

UNIT 3, 3B P ULV E RIZE R

E XCE S S IV E P INION
GE A R FA ILURE S
Foundation
was brok en
E x c ess ive
vibrations &
stres s es
induc ed to
foundations
Im proper
alignm ent between gears
Im proper
ins tallation of
gears
P roc edure was
inc orrec t
Im properly
m anufac tured
gears
P inion/B ull
gears got too
hot
M ec hanic al
failure of
m ounting bolts
Tim ely
c heck of
pinion/bull
gears not
perform ed
Lube oil
sy s tem
failed
Lube oil
pum p failed
C.O.
Im properly
m anufac tured
gears were
k nowingly
ac c epted
C.O.
I.D..
C.O.
C.O. Rev is e
A s s ure
proc edure
f oundation
to ac c ommo-
integrity at
date
w as not per-
eac h pinion
dif f erenc es
f orming
replac ement
betw een
c hec ks .
pulv eriz ers
Maint.
Operarions
began
c hec king
f or s urv iv al.
Now us ing
better
tec hnology .
Res tored
lubeman
pos ition.
Improv ed
ins trumentation.
Figure 4-6
Unit 3 Pulverizer Pinion and Bull Gear Failures, 38,881 MW/Hrs.
4-19

U3 RE HE A TE R
B A CK P A S S
FLY A S H E ROS ION
Unquantified
effec ts of unit
m odific ations
in the boiler
High as h
c ontent
in c oal
B oiler dam per

operation
High flue gas

veloc ities
P oor as h
dis tribution
Furnac e s m all
for heat releas e
required
Reheat &
s uper-heat
outlet
tem perature
c ontrol
Clos e tube
s pac ing in
bac k pas s
A s h m om entum
c onc entrates
partic les in reheat
bac k pas s and S .H.
boundary wall.
B oiler des ign
P oor m ill perform anc e,

finenes s & dis tribution
Fuel level
c ontrol
C.O.
Figure 4-7
Unit 3 Backpass Flyash Erosion 39,378 MW/Hrs.
4-20
P oor m ill
m aintenanc e
C.O.
B all c harge
im proper
C.O.
B oiler dam per

operation
Reheat &
s uper-heat
outlet
tem perature
c ontrol

UNIT 4 AIR
P REHEA TE R
FA ILURE
Cold side
bask ets fell out
B ask et
fas tners
failed
Fram e around
AP H bas ket
failed
Acc elerated
therm al
fatigue of m etal
Therm al fatigue
of m etal.
A PH rotation
speed was set
too low.
C.O.
AP H was being rem oved

from service
on cold s tartups to
place baghouse in
s ervic e earlier.
C.O.
Figure 4-8
Unit 4 Air Preheater Failure 166,744 MW/Hrs.
4-21

Fly as h E ros ion

in B ac kpas s of
Units 4&5
Loc aliz ed high

velocities in
flue gas
High fly as h
levels in
B HP c oal
B HP c oal is
low c os t
option
N.C.
High flue gas

volum es exis t
for boiler
des ign
Flue gas has

poor
dis tribution
Unit overfired
for ex tra M W s
I.D.
N.C.
P erform flue
gas veloc ity
distribution
tes ts
Figure 4-9
Units 4 and 5 Backpass Flyash Erosion
4-22
Ex c es s air us ed
to prevent
pluggage

UNIT 4 A IR
P RE HEA TE R
FA ILURE
Cold s ide
bas k ets fell out
A P H rotation
s peed was s et
too low.
C.O.
B as k et
fas tners
failed
Fram e around
A PH bas k et
failed
Ac c elerated
therm al
fatigue of m etal
Therm al fatigue
of m etal.
A P H was being rem oved

from s ervic e
on c old s tartups to
plac e baghous e in
s ervic e earlier.
C.O.
Figure 4-10
Unit 3 HP Turbine Nozzle Plugging, 100,628 MW/Hrs.
4-23

MW Hour Lost Area
MW/Hrs. Lost
Unit 5 #4 Control Valve
197,471
Unit 4 Air Preheater Failure
166,744
Unit 3 Turbine Copper Fouling
100,628
Unit 4 LP Exciter Field Ground
45,229
Unit 3 Boiler Tube Leaks
39,378
Unit 4 Flyash Erosion
39,288
Unit 4 Battery Failure
38,413
Units3 Pulverizers
34,881
Unit 4 HP Exciter Field Ground
33,500
Unit 3 Scrubber, and Recycle Pumps
33,446
Total
4-24
728,978

UNIT # Type Furnace

1natural circ/front wall fired
4super critical/front & rear fired
5super critical/front & rear fired
Fuel
MFG.
Start up Date Description
Riley Stoker
1962sub-bituminous
Riley Stoker
196320-25 % ash
Foster Wheeler
19649200 btu/ft3
B&W
1969
"
B&W
1970
"
Oper Pressure MW Rating Pollution Control

1800
170wet scrubbers
1800
170wet scrubbers
2000
220wet scrubbers
3500
755baghouses/wet scrubbers
3500
755baghouses/wet scrubbers
net
4-25

CHOLLA PLANT
1999 Lost Generation Incident Review
In an increasingly competitive environment, the ability to reliably produce the lowest cost power
is essential for the long-term survival of a utility. With this vision in mind Arizona Public
Services Company (APSC) requested EPRI to review the 1999 Lost Generation Action Plans of
Cholla Plant. The intent of this review was to provide assurance or not that the true root
causes of lost generation have been identified, and appropriate action plans are in place to
address these causes.
The reviews took place between 19 June to 30 June 2000 at the Cholla Plant. Eleven incidents
were reviewed, of which three had some documentation backing up the plants root cause
analyses. The review team, two from EPRI and one from Cholla and one from Four Corners
used a series of interviews with staff to re-construct the incidents. A number of staff involved
with the incidents were interviewed. Two methods, Behavior Justification and Event and Causal
Factor Charting were used to re-construct the incidents. Diagrams/charts were constructed from
information that depended, almost entirely, on the interviewees long-term recollection of the
incidents. Verification of information was difficult, as there was a lack of formal, incidentspecific documentation and because of the time that had elapsed between the incidents and the
analyses. In many incidences, perceptions and memory of what happened, and in what sequence,
varied, emphasizing the importance of performing root cause analysis in a timely manner, and
documenting the results.
Based on the information obtained during the two weeks at the plant, the EPRI team uncovered
many opportunities for improvements in operations, equipment maintenance, inter-departmental
communications at all levels of management, human performance, root cause documentation and
engineering response. The Lost Generation Action Plans as developed by the plant need these
improvements to ensure that they have a sound base. Documentation of incident investigations,
no matter how minor, provide an audit trail to ensure a historical record of what has occurred at
the plant, how it was analyzed, what the causes of the problem were and how they were solved.
Lack of this audit trail often leads to repeat incidents because staff are not aware of what has
happened previously/how problems were solved. This often leads to short-term, inadequate
solutions, rather than structured, long-term and cost-effective solutions.
No formal structured root cause analysis process exists. Some plant employees have been
trained in a root cause analysis technique, but its use is limited and the results are not
documented. The staff members interviewed were all cooperative, forthcoming and interested in
using a root cause analysis tool to solve long-term problems at the plant.
Each incident reviewed has a number of recommendations. These recommendations need to be
evaluated and implemented. The main recommendation is that a formal structured approach to
incident investigation needs be developed and implemented. A corporate directive should be
developed to give guidance on the requirements. As a start all unit trips/forced outages, load loss
4-26

and failures to meet expected output should be subjected to thorough root cause analysis.
Incidents where plant damage has occurred should also be investigated and any significant
capital expenditure needs to be backed up with a justification that is based on a thorough root
cause analysis. Training a Root Cause Analysis Technique needs to be implemented.
4.10 Introduction
In an ongoing effort to improve reliability and reduce maintenance costs in the APS system,
EPRI was contracted to review the 1999 Loss Generation Action Plans at both the Cholla and
Four Corners plants. The main objectives of this review were:
4.10.1 To provide assurances that the true root causes of lost generation have been identified,
and appropriate action plans are in place to address these causes:
4.10.1.1 determine and verify the true root causes of technical and/or production
problems, and
4.10.1.2 identify preventive actions to effectively manage the identified problems
and minimize the affect on availability.
The team consisted of, two from EPRI and two from APSC, one from Cholla Jack Stant and
one from Four Corners Dan Wilson. Jack Stant Cholla participated in both reviews. The
review took place between June 19 though June 30 at the Cholla Plant. Over fifteen staff
members were interviewed regarding the ten incidents reviewed. These incidents represented a
loss of 481,455 Mwh. The incidents selected represented the major Mwh losses and were
prioritized by the respective plant management teams. The selection was based on their
significance, available documentation, and availability of staff having knowledge of the
incidents.
The Behavioral Justification method of analysis was used to analyze a number of the incidents APS personnel have been trained in this method. Three of the ten incidents had adequate
documentation.
Events and Casual Factors (E&CF) Charting was used as an alternative method. In this method
the sequence of events is charted chronologically, from left to right, and information/conditions
relating to events in the sequence are charted directly underneath the events. Events are shown
as text within rectangles, joined by arrows, ending in a circled unwanted event (reason for the
analysis). The information/conditions are shown in ovals, joined by lines that show their
association. Dotted ovals or rectangles indicate information that still needs to be verified.
Causal factors are identified by asking the question Which of these factors, if eliminated,
would have either prevented the incident from occurring or at least minimizing its impact?
From these causal factors Root Cause are determined. Root Cause is the cause/s which, if
eliminated, would prevent this event recurring or minimize its impact.
4-27

4.11 Plant Description

4.11.1 The Cholla power plant is situated in NE Arizona and consists of four corner fired units,
manufactured by Combustion Engineering and went into service between 1962 and 1981.
They are all drum units operating at a pressure of 2000-PSI. Unit 1, rated at 110 Mw, has
SO2 scrubbers. Unit 2 rated at 245 Mw is equipped with SO2 scrubbers and an absorber.
Unit 3, with a precipitators and is rated at 260 Mw. Unit 4, rated at 380 Mw, is fitted
with a precipitator and an absorber. Units 1, 2, and 3 have Westinghouse turbine/
generator and Unit 4 has a GE turbine/generator train.
4.12 Incidents Investigated

The incidents selected and the Mwh lost are listed in the following tables:
Table 4-1
Cholla - Megawatt-hour loss summary report-fourth quarter 1999
Megawatt hour loss area
Unit 2 Waterwall Hydrogen Damage
188,944
Unit 4 Scrubber Outage
76,917
Unit 4 Air Pre-heater Blockage
48,740
Unit 4 High Opacity Shutdown
42,731
Unit 1 Mechanical Flyash Collector
41,146
Unit 2 A ID Booster Fan Motor Failure
30,464
Unit 2 Cable Tray Fire
28,899
Unit 1 Turbine Thrust Trips
9,971
Unit 1 ID Booster Fan Bearing Failure
7,544
Unit 4 Air Pre-heater Guide Bearing Fire
6,099
Total
4-28
Mwh lost
481,455

4.12.1 Mechanical Dust Collector Retrofit 41,146 Mwh.

Those interviewed were: El Pahi, Dennis Stewart and Mike Machusak
Problem Description
The original Dust Collector on Unit 1 was becoming too expensive to maintain because of the
increasing number of failures also because of its size, number, and location of tubes it was also
difficult to maintain. The design as such made it difficult to address equipment wear problems.
In 1997, an Engineering Service Request (ESR) was written in order to find a permanent solution
to replace the dust collectors. In order to replace the collector during the March 1998 the
project was placed on a fast track. This was the first opportunity as the unit outage frequency had
been changed from two to three years. This allowed three-months between Request for
Proposals and installation of the equipment. Because of this time constraint, no modeling studies
were required or performed prior to installation.
Performance tests showed that the new collector met opacity requirements, but failed to meet the
pressure drop (dP) specification. The dP was 7 inches versus the 3.5 inches specified. In May of
1999, two modifications based on modeling studies made by the manufacturer were installed on
the dust collector at the vendors expense. Subsequent tests showed slight improvement to the
dP, but it still did not meet guarantee. The fans could not accommodate the additional pressure
drop. As a consequence the unit was de-rated by 5 Mw.
The E&CF chart in Figure 4-11 identifies a number of causal factors. These are:
Lack of long term planning. The poor performance of the dust collector was known some
time before it was decided to proceed with a retrofit. The Engineering Services Request to
resolve the collector problems was written in late 1997. Shortly there after, the scheduled
outage interval was increased from two years to three years. This increase in frequency
changed the priority to perform the modification during the 1998 Spring outage. Failure to do
so would result in a three year delay of the project, and engineering studies predicted that the
collectors remaining life was less than three years
A bid package was put together using inlet design and pressure drop specifications, and sent
out in January of 1998 to three vendors, with a deadline of three weeks for responses. Due to
time constraints, no modeling studies were required. ABB, the OEM for the collector, did not
respond.
The bids were not thoroughly analyzed and or evaluated. No design review was performed
to validate the suppliers technical guarantees.
Contract management was questionable. After installation, the collector failed acceptance
testing due to a high-pressure drop of seven inches. In May 1999 the contractor modified the
plant based on a modeling study. Test showed a decrease of 1.6 inches in dP to
approximately 5.4 inches against a guarantee of 3.5 inches. This additional dP was greater
4-29

than the capability of the induced fans at full load. This resulted in a 5 Mw de-rate on the
unit. No additional requirements were put onto the contractor.
A feasibility study is being undertaken to either replace the fans, or changing the blade
design to overcome the collector pressure drop and return the unit to full load capability
The above represent the casual factors that if removed would have either prevented this incident
from happening or at least would have minimized its impact. The root causes would need to be
established from these causal factors by answering the question What management processes
should have been in place and enforced to ensure timely and cost effective project management?
4.12.2 Unit One ID Booster Fan Bearing Failures 7,544 Mwh.
No documentation was provided however El Pahi and Jim Fogel gave some background to this
bearing failure.
Repeat booster fan bearing failures have occurred. The bearing are water-cooled sleeve
bearings. The failure mode has generally been overheating caused by lack of adequate
lubrication. The ash-laden gas that is conveyed by the fan erodes the fan blades causing an
imbalance in the fan rotor. This imbalance results in increased vibration. The vibration is
transmitted through the bearing housing to the cooling water supply housing, cracking the inlet
nipple. This allows water to displace the oil in the bearing sleeve, causing the bearing over heat
and fail.
Root cause Analysis Chart
Figure 4-12 shows the E&CF chart drawn from the information obtain from the interviews.
Plant personnel assumed that the increased fan vibration due to flyash erosion resulted in
accelerated metal fatigue of the cooling water supply nipple. The following actions have been
taken to resolve the problem:
Cooling water has been replaced with an air- cooled system.
In 1998 the worn/damaged bearing housing on A fan was replaced and the fan shaft
repaired.
A newly installed vibration monitoring system now trips the fan at seven mils.
Installation of an external lube oil-cooling system.
The above actions have so far halted the failures. However, these actions only address the
symptoms and/or give additional protection. None of the above address the causes, which are
vibration, caused by fan imbalances, caused by erosion caused by excessive ash-laden gas.
4-30

4.12.3 Unit one Turbine Thrust Trips 9,971 Mwh.

Limited documentation was made available. George Ross and Tim Vachon provided some
background to this incident.
One thrust trip setting was desired for all units at the plant. However, Unit 1 Turbine always ran
with a higher thrust of plus or minus 10 mils compared to an expected value of plus or minus 6
mils. This was aggravated by the fact that the Bentley system and manual methods did not agree
on thrust position. This resulted in numerous spurious trips of the Unit 1 turbine.
During the spring 1998 outage, the abnormally worn thrust bearing was replaced. The turbine
was thoroughly inspected, and all possible contributors to the abnormal thrust were examined.
New thrust probes were also installed and the thrust was statically calibrated. 2 mils of thrust
were available for adjustment. On returned to service, the unit tripped on high thrust. For some
unknown reason the thrust zero point had moved to -15 mils from the cold set point. While off
load the probes were adjusted and the unit returned to service. The unit again tripped on high
thrust. While off load the probes were again adjusted. Several trips occurred and adjustments
made to the trip limit until a trip setting was obtained where the unit did not trip on high thrust.
The probes could not be adjusted online. The above actions have eliminated the trips however
the turbine now runs with a thrust value of between 12 to 15 mils which is a concern to plant
staff.
The E&CF chart in Figure 4-13 identifies the causal factors for the repeated trips after the new
probes were fitted. The reason for the high thrust in the first place still needs to be established.
There are three causal factors identified.
Incorrect calibration of the newly installed probes. Inexperience/lack of know how and
insufficient data are the main reason for the incorrect calibration.
The decision to limit the meter range in order to maintain plant standards. Since the cause of
the excessive thrust had not been identified, a conservative approach was taken to protect the
turbine from damage.
The inability to adjust the probes while the turbine is online. Conservative adjustments were
made to the thrust trip set points each time the turbine tripped since a large adjustment could
have had serious consequences.
Although the trips have been eliminated, there is concern over the turbines excessive thrust
readings. The investigation is ongoing and listed in the Lost Megawatt Action Plan. Those
interviewed revealed an in-depth knowledge of the problem and are committed to its elimination.
4-31

4.12.4 Unit Two A Booster ID Fan Motor Failure 30,464 Mwh.

Those interviewed were: Andre Norwood, Lee Holly, Jim Wilber, Allen Bushman, Jack Stant
and Joe Lacey
In June of 1999 two attempts were made to start Unit 2 A Booster ID fan following a planned
unit outage. Both attempts resulted in an instantaneous ground fault. An investigation revealed
that a power supply cable that had been disconnected for testing purposes during the outage had
not been reconnected. The fan motor insulation was tested acceptable after reconnection of the
cable and placed in service. While in service the vibration level increased. On July 11, the unit
was returning to service after a tube leak repair and the fan was started six times for washing and
balancing. On the sixth start the motor failed and burned-out. The badly damaged motor sent
off site for repair.
The E&CF chart in Figure 4-14 shows the sequence of events. The analysis of this incident was
difficult and required extensive interviews and review of available documentation. The following
needs to be read in conjunction with the events mapped in Figure 4-14
During the 1999 spring outage, APS Generation Engineering personnel performed extensive
Doble testing on the Units large electrical equipment. Test results on the A booster fan
motor showed some deterioration of the motor cables. In order to quantify the problem, the
testing procedure was altered. This consisted of disconnecting the cables at the breaker panel,
which is not normally done. No one in the plant was informed of this change in the scope of
work. Manpower scheduling problems necessitated using four different contract electricians
on the job. The breaker cabinet was returned to service with the middle phase disconnected.
The Doble test showed that the cable had deteriorated and needed to be replaced during the
next planned outage.
On June 18, operations attempted to start the fan for balancing. The fan tripped on
instantaneous ground fault. No evidence of damage could be found. The protection flags
were reset and a second attempt was made to start the fan. This time it tripped on overload.
The production supervisor contacted electrical maintenance personnel to investigate the fan
failure. A motor insulation test indicated that the motor had an open phase. At this point, the
Outage contract electrical foreman, who was still on site, remembered that the cables had
been disconnected during the outage. It was then discovered that the cable was still
disconnected. After returning the cables to their proper configuration, the motor was tested
and found acceptable, and the fan returned to service. No Doble testing was performed on the
motor or the cable. Fan balance and cleanliness was verified.
The fan operated for seventeen days with no problems. On July 6, the fan inboard bearing
vibration increased to 3.5 mils horizontal and 2.5 mils vertical. The outboard bearing
remained constant at 1.9. Plant personnel assumed that fan vibration was the result of
deposition of material on the fan blades as a result of scrubber de-mister failure. However,
4-32

the de-mister problem had been addressed during the spring outage and appeared to be
functioning properly. There was no detailed analysis done on the increasing vibration.
On July 10, the unit was shut down for tube leak repairs, and the fans removed from service
once the boiler was cooled. On July 11, based on an increase in vibration levels, the fan was
washed. The fan was started six times between 19:20 and 00:06, a period of four hours and
fourteen minutes. Analysis of the start and stop times table in figure 4-14 shows that the
plant was in compliance with the industry standard of not starting a motor more than three
times in any one hour period. From previous washing experience a decrease in the vibration
levels is noticed after the first washing cycle. This was not the case with this fan - washing
of the blades had negligible effect on vibration levels. On the sixth start, the motor failed
immediately.
The following represents the causal factors i.e. those causal factors that if were not present would
have either prevented this incident from occurring or at least minimized its impact.
The original Doble testing work scope was expanded without notifying the Plant Clearance
Authority. Because of this, no one on the plant staff was aware that the cables had been
disconnected for testing.
The two ground faults may have significantly contributed to the failure. However, this cannot
be accurately determined as the motor was repaired without any failure analysis being done
on it.
Doble testing was not performed on the fan motor following the two unsuccessful fan starts
on June 18. Doble testing is a more rigorous test and may have revealed deeper problems.
No analysis of the increased vibration levels. It was assumed that ash was lodged on the fan
blades. This assumption was made in spite of the fact that the de-mister problem had been
corrected, and the vibration increase was on the inboard bearing only.
Excessive fan starts. A careful look at the fan start timetable shows that the production
supervisor was following the guidelines regarding the time limit on fan starts. This guideline
is part of the plants oral history and could not be found in any written procedures.
4.12.5 Unit Four Air Pre-heater Pluggage 48,740 Mwh.

The following were interviewed - Layne Miller, Jack Stant, Pete Beltran and Mark Soloman
Problem Description
In September of 1999, Unit 4 was removed from service due to a blocked air pre-heater. The first
indications of blockage were observed in late 1998 Plant instrumentation and alarms indicating
blockage were either ignored or misdiagnosed by both operations and maintenance staff. The
blockage continued unabated until the unit was force to be taken out of service. During this
period two opportunities to inspect the air pre-heater and or check instrumentation were missed.
An inspection of the air pre-heater revealed that the sootblower nozzles had broken off,
rendering the sootblower system ineffective.
4-33


The E&CF chart in Figure 4-15 identifies a number of causal factors resulting in this incident.
In mid-1998 no investigation was undertaken or initiated when plant personnel noticed that
no opacity spikes occurred during air pre-heater sootblowing. This is a normal occurrence
when sootblowing the air pre-heater in operation. An investigation would have lead to the
realization that the air pre-heater was blocking up.
In October of 1998, operations wrote a Work Order on high dP on Unit 4 air pre-heater B
wheel. Maintenance diagnosed it as a transmitter problem.
In February of 1999, operations wrote another Work Order on high dP on both A and B
wheels of Unit Four air pre-heater, specifying potential blockage of the air pre-heater. Again,
the diagnosis was faulty transmitters.
Plant engineers do not trend or review data on the pre-heater. Trending the dP would have
highlighted the blockage earlier.
The air pre-heater soot blowing nozzles have been added to the units inspection routine. It is
also planned to install pre heater airside soot blowers. This will solve the mechanical problems
identified in this root cause analysis. However, the habit of ignoring or not repairing
instrumentation with known problems (see Figures 4-17 and 4-19) is a serious trend that needs
to be addressed.
4.12.6 Unit Four Scrubber Problems 76,917 Mwh.
Those interviewed were Jack Stant and Pat Reynolds
Problem Description
Unit 4 stack SO2 levels were increasing to unacceptable levels. Examination of operating
parameters showed low flue gas flow through the scrubbers. The high dP across the de-misters
confirmed that some blockage was taking place. During the spring 1999 planned outage, Unit 4
scrubber was inspected to gather data to be used for analysis. In May of 1999 the unit was
removed from service and extensive repairs and modifications made to the scrubber. In the
spring of 2000, proper nozzles were installed on Unit 4. Post outage SO2 and dP are within
limits.
Figure 4-16 represents the analysis completed by plant staff using the Behavior Justification
Root Cause Analysis methodology. All three identified correctable opportunities were addressed
during the outages.
The de-misters were replaced with a type less prone to plugging.
Correctly sized nozzles were installed.
4-34

The control scheme for spraying was improved in order to give Operations control over
spraying frequency and duration.
This analysis was simple, but effective in identifying the problems and solutions before shutting
down the unit for repairs. Advantage was taken of opportunities to inspect the equipment for
further information. The approach was well planned and coordinated. This is an example of how
plant problems/incidents should be managed.
4.12.7 Unit Four High Opacity Shutdown 42,731
Those interviewed were: Jack Stant, Elbert Watts, Junior Ward, Layne Miller, Mark McKenzie,
Pete Beltran and Mark Soloman
Problem Description
Due to a strike at the McKinley mine, coal was purchased from an alternate supplier. On 5 June
2000, 5,500 tons of this coal was bunkered to Unit 4. Unknown to plant staff, the coal was a 30%
ash coal instead of the expected 22%. The fly ash removal system was unable to cope with the
increased ash loading, and Transformer Rectifier (RTs) sets in the precipitator began to short out
because of high ash hopper levels. Unit load was decreased to control opacity, but was
ineffective and opacity reading continued to increase. On June 11 the unit was taken off load to
prevent further violations of opacity limits.
Although this event was not in the original scope of work, it was included at the request of
Cholla plant management. An E&CF chart was drawn. Figure 4-17 shows the charted sequence
of events. Four causal factors were identified
The coal quality directive as written did not address the ash content of the coal received.
APS purchased coal from an alternate mine that had previously supplied coal. Ash content is
not addressed in the specifications, however the mine informed APS that coal in storage had
an ash content no greater than 22% in fact the coal contained 30 % ash.
The practice of verifying ash hopper empty was no longer done. It was customary for
operators to check hopper level prior to starting the emptying sequence. This practice had
not been passed onto the new operators and consequently did not get done on a routine basis.
Hopper level indication was inoperative. The level indicator were considered unreliable and
therefore were neglected.
Preventive maintenance on the Nuva Feeders had been suspended and as a result
numerous failure occurred simultaneously, which could not be kept up with.
4.12.8 Unit Two Waterwall Tubes Hydrogen Damage 188,944 Mwh.

Those interviewed were: Dan Nass and Kim Belknap
4-35

Problem Description
In the early nineties several poor quality water excursions, caused by condenser tube leaks,
occurred on Unit 2. This, coupled with a lack of chemical cleaning, resulted in internal tube
scaling - hydrogen damage. In 1999 fifteen tube failures occurred, of which, eleven were
identified as Hydrogen Damage.
Figure 4-18 represents the root cause analysis completed using the Behavior Justification
methodology.
The unit was operated with high chlorides, which entered the boiler via condenser leaks.
Poor control of the demineralizer regeneration process contaminated the makeup water with
acid, reducing boiler pH. The boiler was operated for hours with a pH of seven.
The practice of draining or chemically cleaning the boiler following severe pH excursions
was not followed.
The unit operated with high dissolved oxygen, producing high levels of corrosion products in
the feedwater system, which eventually deposited on waterwall tubes.
The chemical cleaning in 1997 was the first in eleven years.
The effects of the poor chemical control in the early nineties was felt in 1996 when the unit
began experiencing tube failures due to hydrogen damage. Appropriate action has and is being
taken by the plant staff. The condenser has been replaced. Makeup water supply has been
contracted out and is of a good quality. EPRI guidelines are followed and the unit chemistry has
been improved. Waterwall panels have been replaced. Management has placed a high priority on
maintaining an effective boiler water quality control program.
4.12.9 Unit Two Cable Tray Fire 28,899 Mwh.
Those interviewed were: Jack Stant, Pete Beltran and Walt McMillan
Problem Description
In March of 1999, scaffolding boards lying on top of a cable tray located above a pulverizer hot
air duct ignited. The fire melted the leads to the Unit 2 service water pumps, which supply
cooling water to Units 2 and 3 soot blower air compressors. The air compressors had not tripped
in spite of high temperatures and were manually removed from service by the Shift Supervisor to
prevent further damage. The Shift Supervisor ordered Unit 2 to be shut down and load on Units 3
and 4 to be reduced. While lowering load on Unit 4, it tripped off line due high furnace pressure
resulting from damper-control low air pressure. Extensive damage was done to Unit 2 soot
blower air compressor because the wires on the high temperature trip were shorted out and not
repaired since replacement was being considered.
4-36


The Behavior Justification analysis figure 4-19 performed by plant staff in 1999 was reviewed.
The E&CF charting method figure 4-20 was also used as a comparison. Both analyses are
lengthy and contain a number of causal factors. The following discussion will be based on the
E&CF chart analysis.
Scaffolding boards had been left on top of the pulverizer hot air duct for an unknown period
of time. The duct conveys combustion air at a temperature of 500 degrees F to the
pulverizers. The location is near the coal transfer point and coal dust settles onto the
ductwork. The area is enclosed.
The fire melted the leads to both service water pumps, which led to loss of cooling water to
the soot blowing air compressors (SBAC). Unit 2 service water pumps provide cooling water
to both Units 2 and 3, as the Unit 3 pumps had been abandoned for ten years.
The Unit 4 Shift Supervisor discovered the loss of service water cooling pumps, as the
supervisor for Units 1,2, &3 had assumed the role of Fire Commander. He immediately
ordered the shut down of Unit 2.
The Supervisor then manually took the Unit 2 and 3 SBACs off line. They had not tripped on
high temperature since the high temperature trip wires had shorted and not been repaired.
An alternate source of cooling water, using the fire water system, could not be located
because the operators were not familiar with the plant layout.
Unit 1s soot blowing air system was isolated and load reduced on Units 3 and 4 to
compensate for loss of soot blowing air. Unit 4 SBAC could not be used as it was down for
repairs.
The Assistant Operator (AO) attempted to isolate the Unit 4 air systems but was unable to do
so, as there was no procedure for him to follow, and he was inexperienced. The AO was
manually closing 4B ID Fan damper, but 4A damper had insufficient instrument air pressure
to follow. The resulting high furnace pressure tripped the unit.
An analysis of this incident reveals that if the scaffold plank had not been left on the hot duct,
this incident would not have occurred. Therefore the primary root cause of this incident is poor
housekeeping and safety culture. Eighteen months prior to this incident a similar fire, involving
scaffold boards, had occurred on Unit 4. The other seven causal factors identified contribute to
the loss of Unit 4 and the extensive damage incurred by the air compressors. As a result of the BJ
analysis performed in 1999, an action plan was put together and is reviewed below.
The shorted wires on the air compressors were immediately repaired. The trip system was
replaced.
The Unit 3 service water pump was returned to service. However, the pump supply is from
the Unit 3 cooling tower and is too high a temperature for use as a cooling water source for
the air compressors. The project is now with engineering.
Perform a monthly combustibles inspection of the units. This was done monthly in the
second half of 1999, but has only been done once in 2000.
4-37

Write a procedure for isolating Unit 4 from Units 1,2, & 3. A rough draft was written in
March of 1999. As of June 2000, no further action has been taken.
The Unit 4 ID fan damper drives have been replaced with Jordan drives, and no longer
depends on instrument air.
In order to ensure a 30-day turn-around on sootblower air compressor overhauls, rebuild kits
have been purchased.
4.12.10 Unit Four Air Pre-heater Guide Bearing Fire 6,099 Mwh.
Those interviewed were: Jack Stant, Dan Hyde, Pete Beltran, Walt McMillan and Bryant
Peterson
Problem Description
Unit 4 had been on line 27 hours following a unit shutdown to perform an air heater wash when a
high temperature alarm on the 4A Air Pre-heater guide bearing in the control room was received.
The AO who investigated the alarm reported a fire and the fire alarm sounded. The Shift
supervisor and AO attempted, five times, to extinguish the flames. At this point, the SS ordered
that the shut down of the unit. The Joseph City fire department arrived on site but could not
make radio contact with the Supervisor. Members of the station fire team arrived and put the fire
out. The bearing was visually inspected and found fit for purpose. The unit was returned to
service.
Root cause Analysis Chart
Figure 4-21 shows the E&CF chart used to analyze this incident.
The high bearing temperature alarms, which prompted the AO to inspect the guide bearing,
was the result of fire in the vicinity to the thermocouple.
The area around both A and B guide bearings is packed with about ten inches of flyash,
which on its own, is not considered to be a fire hazard. Over a period of time, the flyash had
become soaked with bearing oil from oil spillage and leaks.
Three conditions are necessary for a fire.
Fuel -The oil, which had soaked the flyash, had a flash point of 440 degrees F.
Temperature- the air passing through is 560-580 degrees F.
Oxygen- The SS reported that the flames were coming from up under the guide bearing,
which was covered with flyash. Only when the nozzle of the fire extinguisher was inserted all
the way into the flyash was the fire put out.
A plant walk down by the review team revealed the exact same conditions, which had led to the
fire, still existed. In an attempt to determine the temperature of the flyash, it was discovered that
4-38

the bearing seal had failed and hot gas was being blown out, and was providing the heat as well
as the oxygen source.
Approximately fifteen feet from the bearing was a drum of oil, two buckets containing oil, oil
soaked rag and gloves. As discussed in incident 4.9, poor housekeeping is the root cause of this
incident. The fact that the exact same conditions still existed, a year later, shows a lack of follow
through/follow-up by supervisory staff and the Fire Chief as they were given that responsibility.
4.13 Findings
At each of the stations, the review team consisted of the two EPRI employees and a staff
member from the other station involved in this exercise. This was a very positive aspect of
the review as experiences and problem solutions at both stations were shared. The
opinions/ideas of peers were welcomed.
Root Cause Analysis is done by exception, i.e. as and when requested by management.
Based on the information provided, Cholla management requested more analyses than Four
Corners. These are undertaken by the Operation Technical Advisor Jack Stant
Many staff members were trained in the Root Cause Analyses technique taught by Bobby
Jones and Associates Behavioral Justification, but they dont regularly use any formal
analysis technique.
Root Cause analysis, commonly, consists of group brainstorming sessions, where solutions
(often short-term) are formulated and actions allocated.
Problem solutions are not well documented and in some case not documented at all.
Allocated actions/solutions to brain storming analysis are rarely followed up
There was very little documentation available from which the review team could gather
information. Therefore, in an attempt to reconstruct the incidents key information had to be
obtained from staff members who had knowledge thereof. E&CF charts were constructed
from this information and causal factor determined.
Of the ten lost generation incidents reviewed, three were adequately documented using the
Behavioral Justification technique.
In more than 50 % of the reviewed incidents, it was found that staff chose to rather
replace/fix/modify equipment or plant, instead of assessing root causes.
In almost all the incidents reviewed /reconstructed, the fix repair/replace or modification
did align with those recorded in the 1999 Lost Generation Action Plan.
Preventive/Predictive Maintenance schedules/activities are not implemented, as staff are

focused on corrective maintenance, as in crisis management.
There is a lack of problem ownership, as responsibility for resolution of problems is not

specifically allocated, e.g. Cable tray/duct fires.
Project kick-off and exit meetings were held at both plants. At Cholla the management team
showed a keen interest and participated.
4-39

4.14 Conclusions
The following conclusions have been derived from the Findings above:
4.14.1 Formal documented Root Cause Analysis is not being done on all major incidents. The
most common reason for not doing a root cause was, we are too busy putting out fires.
This made it difficult to verify the actions identified in the 1999 lost Generation Action
Plan.
4.14.2 Although no formal analysis is being done, the actions (solutions) identified in the 1999
Lost Generation Action Plans are considered adequate short-term solutions. Long term
solutions, come from an in-depth root cause analysis of the incident and/or problem.
4.14.3 Knowledge of a Root Cause Analysis process was limited even though staff had been
trained in an analysis technique.
4.15 Recommendations
4.15.1 A formal structured Root Cause Analysis process needs to be implemented. This process
should be developed based on a corporate directive requiring those major incidents
(greater than a $X, 000 value to be determined by Corporate), unit trips and a
significant unexpected maintenance expenditure be investigated to ensure repeat
failures/incidents/trips do not occur.
4.15.2 Staff should be trained/retrained in a Root Cause Analysis technique that includes Human
Performance aspects.
4.15.3 The various recommendations identified in Section 4.4 Root Cause Analysis should be
reviewed, evaluated and implemented.
4.16 Appendix 2
Root Cause Analysis Charts
(Excel file)
Note:
1. A number of the charts are incomplete as there was limited documentation and the allocated
time per incident was constrained. However they do reflect the information given or
presented by the various interviewees.
2. The charts need to be read in conjunction with the text in the main body of the report
4-40

Unit 3
HP Turbine Nozz le
Copper Deposition
Copper in
m ain steam
Copper in
feedwater
Copper in
c ondens ate
Copper
m etallurgy in
c ondenser
C.O.
Condenser
being replac ed with
S .S. 29-4-C
Copper
m etallurgy in
LP feedwater
heaters
C.O.
Copper
m etallurgy in
HP feedwater
heaters
C.O.
Switched from
hy draz ine to
Elim inox in 93
C.O.
Drum pres sure

2250-2300 P S I
N.C.
Returned to
hy draz ine in
early 2000
Copper levels
in E con. Inlet
reduced
Figure 4-11
Unit One, Mechanical Dust Collection Retrofit, 41,146 Mwhs
4-41

High Fan
Vibration
Fan inbalanc e
due to flyash
erosion
Cooling water
s upply nipple
c rac ks
Fatigue
Cooling water
enters lube oil
res ervoir
No analys is
of failed nipple
Rigid
design
Hous ing
worn
Figure 4-12
Unit One, ID Booster Fan Bearing Failures, 7,544 Mwhs
4-42
Cooling water
dis places
lube oil
B earing over
heats .
No hi-hi
bearing
tem perature
trip
Repeated
Boos ter ID
Fan B earing
Failures

1998
Repeated trips
due to high
thrus t
Turbine has
m ore thrus t
range than other
Cholla units
Turbine overhaul
c om ponent
replac em ent
W orn thrus t
bearing
replac ed
P robes
replac ed
Different
thrus t
loc ating
m ec hanis m
Turbine thrus t
m eas ured by
c old s tatic
m ethods
Unit c am e up
on load
New thrus t
probes
ins talled
P robes
c alibrated
bas ed on pas t
ex perienc e
Unit 1
Turbine
Thrus t
Trips .
Turbine thrus t
goes inc reas ingly
negative with
inc reas ed load.
Lac k of
ex perienc e
Turbine ex pands
with inc reas ed
tem perature
Lac k of
data
M eter range
lim ited to m aintain
plant s tandards
Thrus t ex c eeded
trip s et point
No c apability
to adjus t probes
on line
Trip s etpoint
determ ined by
c old s tatic
m eas urem ents
Figure 4-13
Unit One Turbine Thrust Trips, 9,971 Mwhs
4-43

6/18
Unit down for

s c heduled
outage
E lec . E ng.
Doble tested
2A B oos ter
fan m otor
Cable dis c onnec ted

at
break er c abinet for
Dolby tes ting
M otor
c hec k ed
out OK
P lanner not
notified of c hange
in work s c ope
Center phase
c able Dolby
tes ted
m arginal
6/21
V ibration
levels
ac c eptable
7/6-7/8
Inboard
Horiz ontal 3.5
V ertic al 2.5
P roduc tion
S upv.
inves tigates
Dec is ion m ade

to was h fans
Dec is ion bas ed

on as s um ed
fouling of fan
blades
No detail
readings /analy s is
done
Figure 4-14
Unit Two, A Booster ID Fan Motor Failure, 30,464 Mwhs
Fan s tarted for

balanc e
c hec k s
7/11
Fan has repeated
s tarts for was hing
and balanc ing
Fan s tarted
6 tim es , no
proc edures or
guidelines
B alanc ing
c hec k ed out
OK
M egger tes t
indic ates open
phas e
Fan verified c lean

after June outage
Firs t was hing

had negligible
effec t on
vibration levels
Unit 2 A
B oos ter
ID Fan M otor
S horted Out
*
DA TE
M otor
c hec k ed
out OK
E lec tric al forem an

rem em bers that
c able had
been dis c onnec ted
No inves tigation
to determ ine c aus e
Outboard
rem ained
c ons tant at 1.9
4-44
E lec tric ian

m eggered the
m otor & c able
Fan tripped
on overload
Flags were down

and res et
7/10
Unit 2 s hut
down
for boiler tube
repair
Fan inboard
bearing vibration
inc reas es
6/18
Operator again
attem pted to
s tart the fan
Fan tripped on
ins tantaneous
ground
Us ed four
different elec ts .
for the tes ts
Contrac t
elec trician
fails to rec onnec t
c ables
Fan plac ed
in s ervic e, ran
for 17 day s
Operator
attem pts to
s tart fans for
balanc e c hec k
S TA RT
S TOP
E LA P S E D
TIM E
--------------------------------------------------------------------7/10
19:20
19:24
19:54
20:05
34 M in.
20:24
21:11
30
21:22
21:31
58
23:07
23:08
105
7/11
00:06
00:06
59
Tripped

Oc t-98
Opac ity spikes

c aus ed by s oot
blowing s top
Feb-99
Operator observes high dP

on "B " wheel.
(A larm?)
Operator obs erves high dP

on "A " & "B "
wheels .
W .O.
W ritten
C429475
W .O.
W ritten
C432684
Obs erved
by plant
pers onnel
Mar-99
A pr-99
Cable tray
fire outage
40 hours
M iss ed
opportunity
Operator observes high dP

on A & B
wheels .
W .O. W ritten
C434285
Fowarded to
E ngineering
*
*
No W ork
Order written
M aintenance
diagnos ed
inc orrec tly
W .O.
s pec ified
pluggage
M aintenanc e
diagnosed
incorrectly
"Not ins t. problem ,

c hange alarm s etpoint s "
A P H dP data not
reviewed or trended
M aintenance diagnos ed
inc orrec tly
May-99
Unit load
curtailed- ran
out of fan
Unit Outage
for s crubber
repairs 192 hrs .
M is s ed
opportunity
Ins trument
shop verifies
trans mitter
ac c urac y
Unit Four A ir
P reheater
plugged
9/99
Dis c us s ed at
m orning m eeting
Unit Outage
to wash wheels
Operations
trouble shooting
problem
Found A P H S oot
blowing nozz le
fallen off
Figure 4-15
Unit Four, Air Preheater Pluggage, 48,740 Mwh
4-45

Unit 4 stack
SO2 is too
high
Low flue gas

flow through
scrubber
P lugged
dem is ters
reduc ing flow
"A"
Upper dem is ters

are plugged
Too m uch
carryover
from lower
dem is ters
C.O. S ee "A"
Lower dem is ters

are plugged
S praying is
ineffec tive
S praying is
ineffec tive
S pray
frequenc y
is incorrect
C.O.
Incorrec t
nozzles
Inc orrect
nozzles
S pray duration
is too short
C.O. I& S
C.O. I&S
Operator can adjus t to

as low as one m inute
C.O.
Figure 4-16
Unit Four Scrubber Outage, 76,917 Mwh
4-46

5/31
Coal purchas ed
from alternate
supplier
Unit on
line
after tube
leak
6/3
6/5
Opacity
trending
upward. 7%
Un-noticed
by operators
Mc Kinley
mine
on strike
Operations
uninform ed
of high ash
P rec ip not
inspected
during
outage
6/6
Opac ity base
line rise
continues
5,500 tons of
30% as h c oal
bunkered to
Unit 4
Coal quality directive

did not addres s
as h c ontent
27%
opac ity
rule us ed
Coal was
30% as h
Practice of
hopper em pty
verification no
longer done.
P .M.'s not
performed
Nuvafeeders
not adequately
m aintained
Do
Did not m eet

specs
5,000 tons /day

typical burn
Training
Repair
6/8
TR's tripping
on A &B side
6/8
Elect.'s working
on "A " s ide high
hopper
indications
6/8
A/O's begin
to manually
pull hoppers
Obsolete, parts
not available
High level
in fly ash
hoppers
Operators unaware
level alarm inoperative
Operators did not
realize the s ituation
or cons equences
they were fac ing
"A" side
alarms not
working
Not
spiking
Analys is not
received until
after loading.
Internal direc tive

being rewritten
6/7
High hopper
indications on
"B" side
6/11
Unit has
opacity
ex ceedences
Hopper level
indication
is unreliable
PM's
6/11
Decis ion m ade
to shut unit
down
Unit outage
due to high
opacity
Dum ped A 18, 17,

&14, and B18
hoppers to
ground
Hopper
inspec tions
Load is
dropped
Operators
verify ing A side
hoppers
"A " s ide had

m ore
dam age
No load
reduction
cons idered
6/16
6/3
6/5
6/6
6/7
Figure 4-17
Unit Four, High Capacity Shutdown, 42,731 Mwh
4-47

Chloride
c ontam ination
in boiler water
Operating
with
c ondenser
leaks
Contam inates
in
boiler m akeup
water
S
Dem ineraliz er
regeneration
m alfunc tion
Heavy s cale
depos its
on boiler
waterwalls
Lac k of
c hem ical
cleaning to
rem ove
s c ale
Lack of proper
c hem ic al
c leaning
Nonc om plianc e
to proc edure
No procedure
in place
Im purities in
feedwater
s ys tem
M
S
Did not
boros cope
water wall
tubes
Sam ples tak en

from
nonrepresentative
area
Operating with
c ondens er tube
leaks
P oor
pH
c ontrol
Figure 4-18
Unit Two Waterwall Tubes Hydrogen Damage, 188,944 Mwh
4-48
Corros ion in
feedwater
s y stem
Operating
with high
dis s olved
ox y gen
Im purities in
m ak eup
water
Flow
ass is ted
c orrosion

S im ilar inc ident

on U4 earlier
(18 M onths ?)
Unk nown
maintenanc e
ac tivity
performed
Coal dus t
ac c um ulates
on hot air
duc t
AO reports
fire
Inves tigated?
Poor dust
c ontrol on
Fire brigade
res ponds
S .S . orders
Unit 2
s hutdown
S .S . s huts
down U-2
SBAC
Fire in
SW pump
Hot
Plank
U2 bottom
disc harge
enough to
lef t on
as h enc l.
pres sure
smell
at z ero
duc t
transf er
Cooled by
points
Pow er f eeds to
500 degree
SW pumps
gas temp
grounded
s erv ic e
w ater
Fire loop
c ooling H2O
s upply not
identif ied
High temp
trip w ires
shorted
*
High temp
trip w ires
not repaired
S .S . orders
U-1 S B A
is olated & 1A
S B A C s tarted
S .S . orders
U-3 to drop
load
S .S .
s huts down U-3
SBAC
Hot
enough to
s mell
No
No
s oot blow ing
Not
air
air
suc ces sf u l
s erv ic e
U-4 SBA C
w ater
out of
*
pumps
abandoned
s erv ic e
High temp
A O attem pts
to is olate
U-4 air
s y s tem s
soot blow ing
Cooled by
U-3 SW
S .S . orders
U-4 to drop
load
U-4 SBA C
out of
No
servic e
procedures
A O c los es
4B ID Fan
damper
3 to 5
perc ent
Unit 4
trips
Los s of
28,899
MW
hours
High f urnac e
pres sure
Inst. A ir pres sure

too low to open
4 A ID Fan damper
trip w ires
A O new
s horted
Fire loop cooling H2O
to unit
supply not identif ied
Figure 4-19
Unit Two Cable Tray Fire, 28,899 Mwh
4-49

Tw o units of f and
one c urtailed
U-3 c urtailed
U-2 s hut dow n
Unit 4 tripped of f
due
to s af ety
Not able to blow s oot
Unknow n
High f urnac e
Los s of adequate
damage due to
pres s ure
s ootblow ing air to
c able tray f ire
plant
ID
Operator low ered
Units 2,3,&4
SBA C's not running
ID damper 3-5%
i h
A damper w ould
not rais e in A uto
d
A ir pres s ure w as
at
U-4 SBA C not
U-3 SBCA not
running
running
SBA C s hut dow n
U-2 SBA C not running
Operator w as
low ering load
Same as U-2 SBA C
SS s hut of f
f or maintenanc e
Units 2,3,&4
Tie v alv e
SS ins truc ted
SBA C's not
betw een 2/3 and
operator to low er
running
4 w as open
load
See prev ious
Bobby Jones
High temperature
Bearing temperature
trips did not w ork
ov er the high
Operator did not
blow s oot
c los e tie v alv e

C.O. M1,2,3,4
Units 2,3, &4

Los t c ooling w ater
to SBA C
SBA C's not

running
* Cont. in upper left
ID
Did not repair

CO M1,2,3,4
U-3 s erv ic e w ater
U-2 s erv ic e w ater
pumps not
pumps not running
running
Pumps s hut
dow n f or 10
No pow er f eeds
to pumps
y ears
CO M1,2,3,4
Both motors pow er

leads in tray melted
to ground.
Cable tray f ire
ID
Figure 4-20
Unit Two Cable Tray Fire March 1999, 28,899 Mwh
4-50
* Cont. in upper left
Not able to
temperature trip poimt
Wires s horted
or below 50 lbs .

0:18
9/24
U-4 s hut
down for
A H was h
14:46
27-S ep
Norm al
s tart up
High
dP
17:48
28-S ep
A O confirm ed
reports fire at
guide bearing
Hi Tem p
alarm on
4A guide
bearing
U-4 on line
for 27 Hrs
Oil soak ed
fly ash pack ed
around housing
appox . 10" deep
Tem p
spik ed
No work done
on guide
bearing
Fire dam aged

therm oc ouple
Oil Flas h
point 440
F
A ir seal
leak ing
on A side
9/25
15:46
S S c alls
Joe c ity
fire dept
28-S ep
6099
MW
Hours lost
18:48
SS & AO
put out
fire
Joe City
fire brigade
too late
Could not
m ak e radio
content
Tem p. m eas.
@ 420 plus F
above s eals on
A & B
A ir out
tem p 560580 F
1:18
17:56
SS & AO
could not put
out fire
28-S ep
S S takes
unit offline
S S tries five
tim es to put
out fire
Flam es
em anating
from bearing
hous ing
29-S ep
18:56
M aintenance
chec ks guide
bearing
*
$700 labor
c harged to
WO
Fire
dam aged
therm oc ouple
29-S ep
Unit bac k
on line
4A guide
bearing tem p
in alarm
A ir s eal
leak ing on A
guide bearing
Guide bearing
ins pected and
found to be OK
Figure 4-21
Unit Four Air Preheater Guide Bearing Fire, 6,099 Mwh
4-51

Problem Description
Unit 1 Mechanical Flyash Collector
Unit 1 ID Booster Fan Bearing Failure
Unit 1 Turbine Thrust Trips
Unit 2 A ID Booster Fan Motor Failure
Unit 4 Air Preheater Pluggage
Unit 4 Scrubber Outage
Unit 4 High Opacity Shutdown
Unit 2 Waterwall Hydrogen Damage
Unit 2 Cable Tray Fire
Unit 4 Air Preheater Guide Bearing Fire
4-52
Root Cause MW/Hrs. Lost

4.1
4.2
4.3
9971
4.4
30,464
4.5
4.6
76,917
4.7
4.8
188,944
4.9
28,899
4.10
6,099
5
PROGRAM AUDIT AT HAWAIIAN ELECTRIC COMPANY
5.1 EXECUTIVE SUMMARY

As a result of Hawaiian Electric Corporation embarking on a Remaining Useful Life and
Generation Asset Management program in 1999, and including in the project the Boiler
Reliability Optimization Program, EPRI was requested to provide an independent review of this
area of the project.
During week 18 June 2001, a series of interviews and meetings were held with a number of
HECO personnel and a number of documents were reviewed. The EPRI team found that progress
had been made in successfully implementing a number of planned activities. The team also
identified a number of opportunities where the current efforts could be enhanced and which,
when implemented, would give additional assurance that the goals set for availability and
reliability would be achieved. These opportunities are highlighted in the main portion of the
report.
5.2 Introduction
In 1999 Hawaiian Electric Company Inc. (HECO) embarked on a Remaining Useful Life and
Generation Asset Management program, to enable all the HECO generating plant to continue to
operate well beyond the 20-year horizon. One of the projects in this program was the Boiler
Reliability Optimization Program. The objectives of the Boiler Reliability Optimization
Program are:
To reduce short- and long-term boiler maintenance costs, while maintaining or improving
overall boiler reliability.
To develop and implement boiler maintenance strategies using EPRIs Streamlined

Reliability Centered Maintenance process.
The project was started in the last quarter of 1999 with a Boiler Tube Reduction and Cycle
Chemistry Improvement training program, then continued in 2000 with a pilot Streamlined
Reliability Centered Maintenance project, which is currently being implemented. Progress has
been made in implementing many of the actions, along with many other initiatives as part of the
overall RUL/GAM program.
The program had reached a point where it was expeditious to have an independent review of the
current project status and plan to give assurance that the goals set would be reached. EPRI was
5-1

Boiler Reliability Optimization Project Program Audit at Hawaiian Electric Company
requested to undertake such a review. The review would give management the assurance that the
overall RUL/GAM program and Boiler Reliability Optimization project goals will be met.
The specific objectives of this review are:
To review the current Boiler Reliability project action plan
To identify cost effective improvements; and
To recommend changes in their action plan to address the identified gaps.
5.3 Findings
5.3.1 Boiler Reliability Optimization Project Plan
There are basic plans, outlined in the PDM Team Strategic Plan 2001 and the Boiler
Reliability Optimization Program Envisioned Role of BRO Team Members, but they differ.
The plans are fairly loosely structured and therefore lack focus, giving little direction for the
development of specific action plans. There is no overall strategy with key deliverables and
dates.
Roles and responsibilities of the key players are not clearly defined. This results in
misunderstandings and confusion about who should be doing what and when. This impacts
the cost effectiveness of this program.
An engineer was specifically employed to assist with the implementation of the Boiler
Reliability Optimization Program. His energies have been redirected into solving a variety of
other technical problems, and he has, therefore, been unable to devote the necessary time and
attention to his primary role. This has occurred because of the above-mentioned lack of
direction regarding roles and responsibilities.
Boiler performance goals tube leaks are outlined in the Boiler Tube Failure
Reduction/Cycle Chemistry Improvement Program Management Mandate. Short-term goals,
as outlined, have only been partially achieved. Long term goals may be achieved but are
aggressive.
The Boiler Reliability Optimization Strategic Plan that is being followed (shown in Appendix
1), shows that a number of deadlines have not been met, or have been extended. The plan
has been filled out with the relevant information regarding the current progress and updated
deadlines.
5.3.2 Project Activities

5.3.2.1 BTFR/CCI Program
5-2
A BTFR/CCI Improvement Program Management Mandate was issued in December 1999,

after the BTFR/CCI training was completed. The mandate outlines the ground rules and

guidelines for the implementation of a Boiler Tube Failure Reduction Program. This gives
direction and focus to the efforts of the core Boiler Tube Failure Reduction team.
This Management Mandate has an annual review cycle. To date this document has not been
reviewed and/or revised.
Some of the short- and long-term goals outlined in the Mandate have been achieved.
As a result of the Mandate being issued, a comprehensive, standardized Boiler Tube Failure
report format was developed, and is being followed.
The Boiler Maintenance Workstation (BMW) program, which is a Windows-based software

program used to track and trend boiler tube failure data and information, was purchased some
time ago. Software compatibility problems have now been solved and the database is being
populated with drawings and tube leak data. Initial training on this program was given to a
selected group some years ago, and no other employees have been trained since, so
knowledge of its true benefits and uses is restricted.
In excess of $3million has been budgeted for the procurement and installation of core water
and steam chemistry instrumentation. Phase 1, consisting of 3 instruments per boiler is to be
installed by the end of August 2001 for Kahe units. Installations of phase 1 instruments are
scheduled for completion by 12/31/01 and 3/31/02 for Waiau and Honolulu respectively.
This will enhance the control of water chemistry, which will minimize boiler tube leaks that
result from poor water chemistry, improving reliability.
5.3.2.2 Streamlined Reliability Centered Maintenance
A pilot project on three systems Boiler Air and Gas, condensate and Feedwater at two
sites, has been successfully completed. The resultant maintenance tasks have been accepted
and are being integrated into the work order management system for Kahe 3. This provides
a basis for cost effective maintenance.
There is no overall preventive/predictive maintenance basis for the boiler pressure parts, as in
tubes. Repairs are carried out as and when failure occurs. This strategy has an impact on the
overall reliability and maintenance costs.
5.3.2.3 Root Cause Analysis
Training in basic Root Cause Analysis was done some time ago, but has not been formally
used in practice besides investigation involving boiler tube failures. Problem solving is done
on an ad hoc basis and repeat failures are occurring, causing problems with reliability and
impacting costs.
The importance of Root Cause Analysis being integrated in the work management process is
not being recognized. It is the most effective means of ensuring improvements in reliability
and saving costs, by preventing/minimizing failures from occurring or recurring.
5-3

5.3.3 Condition based data collection
A hardware breakdown structure does not exist for the boiler i.e. uniquely identifying the
system, sub-systems and components within the boiler. Therefore tracking the condition and
or performance of each system, sub-system is difficult.
Condition based data is collected on some sub-systems and components. However the data
collection-analysis-action process is no consistent across all boiler components. This makes
it difficult to determine the true condition of each sub-system or component and prioritize the
inspection and repair work on the boiler.
5.4 Conclusions
In an effort to provide assurance that HECOs current initiatives in the Boiler Reliability
Optimization project are adequate and cost effective, reviewers from EPRI studied the given
findings and concluded that:
5.4.1 The Boiler Reliability Optimization Team and others associated with reliability
improvements have made a significant effort to implement the boiler tasks associated the
RUL/GAM project. All involved need to be congratulated on their efforts and having the
courage to request that a third party review their program.
5.4.2 Basic project management principals of time, cost and quality have not been followed in
the development of an adequate program and schedule.
5.4.3 Some project planning has been done, but it is ineffective. The effectiveness of the
teams efforts is impaired because of the lack of clarity in the plan on roles and
responsibilities, direction, objectives and deadlines.
5.4.4 There is a lot of emphasis and effort being focused on the Boiler Tube Failure Reduction
program and the program is achieving its objectives.
5.4.5 The current efforts do not provide for an adequate maintenance strategy regarding boiler
tubes.
5.4.6 As root cause analysis has not been rigorously practiced since the training sometime ago,
it is doubtful that a significant proportion of that knowledge and skill gained is available.
5.4.7 Significant financial resources have been allocated to install state of the art water and
steam chemistry instrumentation however little resources have been set aside for the on
going maintenance of these newly installed instruments, some of which will be installed
shortly
5.5 RECOMMENDATIONS
The following recommendations are given to enhance the current Boiler Reliability Optimization
initiatives.
5-4

5.5.1 A project leader should be designated in writing, whose prime responsibility it would be
to oversee the implementation of this project.
5.5.2 Develop a comprehensive plan/schedule for the implementation of all
activities. This plan should identify:
boiler-related
The people responsible for the various activities identified in the program, and should
clearly define their roles.
What these activities are as in scope
The duration of each activity.
The resource loading required to implement these activities
A typical Microsoft Project plan would provide this framework.

5.5.3 The Management Mandate and various other programs mandates/directives (i.e. Root
cause Failure, Streamlined RCM etc) should be reviewed and updated, according to the
given schedule, as they serve as the bases for Boiler Reliability Optimization. A two
yearly review cycle is considered to be the norm.
5.5.4 In order to compliment the current BTFR initiatives, it is recommended that a Failure
Modes and Effects Analysis (FMEA) be done on boiler pressure parts, as a means of
identifying potential failure mechanisms, and the financial effects of their occurrence.
This analysis will also focus the organization on what the most significant failures are, so
that a quantum leap in reliability can be made with fewer organizational resources being
utilized. FMEA is an engineering technique used to define, identify and eliminate known
and/or potential failures, problems, etc. on a system and/or components within that
system. Appendix 2 gives an example of an FMEA analysis.
5.5.5
Continue with the current SRCM initiatives on the remaining systems, i.e. to develop a
cost effective maintenance program, based on system functionality that will enhance
system reliability. This will make optimum use of available maintenance resources and
provide a documented base for future additions/revisions to the maintenance program.
5.5.6
Develop a hardware breakdown structure for each boiler. The structure will allow for
each system, sub-system and component to be uniquely identified. This numbering
system can be used in MIMS, Plant View and BMW thus enabling the condition of each
system, sub-system and component to be tracked and enabling cost effective run-repairreplace decisions to be made.
5.5.7
Clearly define and document the roles and responsibilities of the individuals whose
responsibility is to continuously monitor the overall condition of each boiler system, subsystem and component and to develop a short and long term inspection plan for each
boiler and that of managing the Boiler Reliability Optimization Project.
5.5.8
Training in a suitable Root Cause Analysis technique/process be given to a small number

of individuals. It is suggested that a number of key individuals be trained to become
trainers of a suitable process that meets the needs of HECO. These individuals can
5-5

then develop a Root Cause Analysis Awareness course that can then be given to
engineers, technicians, operators and craftsperson.
5.5.9 An administrative mandate/policy/procedure to be drawn up to: standardize what gets
investigated; outlines the investigation standards and processes to be followed; the roles
and responsibilities of the various team members and the reporting requirements. See
appendix 3 for a typical list of headings that appear in a procedure.
5.5.10 A maintenance strategy be developed and documented in the CMMS system for the new
cycle chemistry instrumentation being installed. The information required to develop the
maintenance requirements can be sort from in house expertise, the OEM and or EPRI.
5-6

Appendix 1: Boiler Reliability Optimization Strorage Plan

HECO BRO IMPLEMTATION PLAN AS PER STRATEGIC PLAN
No.
Activity
Formalize and Organize Unified BRO

Team and Procedures
By Whom
Bill and Bob Isler
By When
Goal
1Q01
Establish Formal Use of BTFR and

RCFA Process for Boiler Tube
Failures
Bill, supported by
BRO Team
Issue BTFR
Procedure Manual
Issue CCI
Procedures Manual
Fully Implement BTFR and
RCFA Process
Formalize Data Collection,

Management and Distribution of
Boiler Information
Bill, supported by
PDM specialist, Ed
Chang
Develop Process Map on

Data Collection and
Storage
3Q01
Assist with Development of Boiler

Start Up Procedures and Combustion
and Flame Management Guidelines
Jose, Lane, Ed,

Bill, Barney and
Russell
Completed Procedures in
place
3Q01
Develop Air Pre-Heater Maintenance

Program
Bill and Larry
Develop Feed-water Heat Exchanger

Maintenance Program
Bill McCraw and

PDM Specialist
Develop Condition
Baseline for all Air
Pre-Heaters.
Plan in place.
Develop Condition
Baseline for All
Heat Exchangers.
Select Mapping
software.
Progress to
date
Draft 90%
complete
3Q01
2Q01
4Q01
Completed
and
established
08/16/01
90%
established
Not yet
formalized
No progress to
date. Still
waiting for
schedule
guidelines
from
Operations
Approximately
40% complete
on baseline
history.
Progress is
subject to unit
outages
Page 1
Revised
completion date
Final Draft
08/31/01
Finalize and
issue 12/31/01
Predict
completion by
12/31/01
Predict
completion by
12/31/01.
Predict selection
by 09/31/01.
2Q01
5-7


No.
Activity
By Whom
Goal
By When
Page 2
Progress to
date
Develop and Implement Lay-up

Procedures for Feedwater Headers
Bill, Barney and

Russell
Refine and Implement

existing Procedures
2Q01
Approximately
60% complete.
Develop and Implement Lay-up

Procedures for Boilers
Bill, Ed, Barney and

Russell
Issue Procedures for the

Cycling of Units
4Q01
Approximately
40% complete.
Install and Implement Nitrogen

Blanketing Systems and all
Feedwater Heaters
Optimize Boiler Chemical Cleaning
Frequency, using Condition Based
Techniques. Oversee the Chemical
Cleaning Process
Develop Strategy and Install all Core
Cycle Chemistry Instrumentation
Bill and Barney
Started 10/00 Complete
4Q02
Waiau by
03/31/02.
Bill, Barney,
Russell and Ed
Develop and have in place

the programs and
Procedures
4Q01
Practice in
place.
2Q01
2Q01
Strategy is
completed and
included in
5year plan.
Develop Plan to Maintain Cycle

Chemistry Instrumentation
Larry and Bill
2Q01
Dormant
waiting for
completion of
Phase I CCI.
10
11
12
5-8
Bill, supported by
Barney, Russell
and P&E
Install the 2000

increment
Develop Strategy
for implementation
for remaining core
instrumentation
Plan to be in place
Revised
completion date
Predict Waiau
Honolulu by
12/31/01.
Predict Waiau
Honolulu by
12/31/01.
.
Formalize
program and
procedures by
12/31/01.
Phase I will be
85% complete
by 12/31/01.


No.
Activity
By Whom
13
Evaluate Cycle Chemistry Procedures

and Software
Bill, Bob Isler,

Barney and Russell
14
Train Operators in Cycle Chemistry

following installation of instruments
Bill, Barney and

Russell
Goal
By When
Select system.
ChemExpert will probably
be chosen
15
Develop and Implement Air Inleakage Improvement Program
PDM Group,
Barney and Russell
Develop Training
Program
Take the lead in
Implementing
Training
Implement Formal Program
Page 3
Progress to
date
Revised
completion date
3Q01
Dormant
waiting for
EPRI Solutions
to convert
current
software to
Windows NT.
(expected by
08/31/01).
Predict system
selection by
12/31/01.
4Q01
1Q02
No progress to
date. Doubtful
will meet
schedule
completion.
2Q01
Practice in
place. Formal
program not yet
written..
Predict 2Q02
5-9

Appendix 2
FMEA Worksheet
System: Heat recovery
Sub-system: Economizer
Component: Economizer inlet header
Causes of Failure
If detected during a
periodic inspection,
repairs would necessitate
extending the outage. If,
however the cracks went
undetected failure would
result in a forced outage
with the consequent loss
of revenue.
1. Start up or shut down

procedure not being
followed
2. No start-up or shut
down procedure available
or
3. Start-up and or shut

down procedures is
inadequate and highlights
no precautions.
Failure mechanism
Cracks on the internal
diameter, parallel to
the stub tube axis
Thermal fatigue
5-10
Detection
Method
RPN
Recommended Action
128
Conduct periodic
reviews/audits of
compliance to the start-up
and or shut down
procedures
256
128
Compile separately a
start-up and shut
procedure detailing the
steps and precautions to
be taken by the operator
Conduct periodic
reviews/audits of
compliance to the start-up
and or shut down
procedures
Detection
(1- 10)
Effects of failure
Severity
(1 10)
Failure Modes/
Frequency
(1 10)
FMEA Process

Appendix 3
List of Typical headings that appear in a policy and or a procedure
1. Group name
2. Document type: policy, standard, guideline, procedure etc
3. Reference number
4. Revision number
5. Description of revisions made
6. Review date
7. Date authorized
8. Date when document came into effect
9. Compiled by
10. List of designations who have approved the document
11. Authorized by
12. Title
13. Background/preamble/introduction
14. Purpose
15. Scope of document
16. References
17. Definitions and or abbreviations
18. Responsibilities
19. Responsibility matrix
20. Requirements: step by step of what is required to be done
21. Work process flow and responsibility matrix
22. Distribution list
23. Appendices
5-11
Target:
Predictive Maintenance Program Development and
Diagnostic Tools
SINGLE USER LICENSE AGREEMENT

THIS IS A LEGALLY BINDING AGREEMENT BETWEEN YOU AND THE ELECTRIC POWER RESEARCH
INSTITUTE, INC. (EPRI). PLEASE READ IT CAREFULLY BEFORE REMOVING THE WRAPPING MATERIAL.
BY OPENING THIS SEALED PACKAGE YOU ARE AGREEING TO THE TERMS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO
THE TERMS OF THIS AGREEMENT,PROMPTLY RETURN THE UNOPENED PACKAGE TO EPRI AND THE PURCHASE PRICE WILL
BE REFUNDED.
1. GRANT OF LICENSE
EPRI grants you the nonexclusive and nontransferable right during the term of this agreement to use this package only for your own
benefit and the benefit of your organization.This means that the following may use this package: (I) your company (at any site owned
or operated by your company); (II) its subsidiaries or other related entities; and (III) a consultant to your company or related entities,
if the consultant has entered into a contract agreeing not to disclose the package outside of its organization or to use the package for
its own benefit or the benefit of any party other than your company.
This shrink-wrap license agreement is subordinate to the terms of the Master Utility License Agreement between most U.S. EPRI
member utilities and EPRI.Any EPRI member utility that does not have a Master Utility License Agreement may get one on request.
About EPRI
EPRI creates science and technology solutions for
the global energy and energy services industry. U.S.
electric utilities established the Electric Power
Research Institute in 1973 as a nonprofit research
consortium for the benefit of utility members, their
customers, and society. Now known simply as EPRI,
the company provides a wide range of innovative
products and services to more than 1000 energyrelated organizations in 40 countries. EPRIs
multidisciplinary team of scientists and engineers
draws on a worldwide network of technical and
business expertise to help solve todays toughest
energy and environmental problems.
EPRI. Electrify the World
2. COPYRIGHT
This package, including the information contained in it, is either licensed to EPRI or owned by EPRI and is protected by United States
and international copyright laws. You may not, without the prior written permission of EPRI, reproduce, translate or modify this
package, in any form, in whole or in part, or prepare any derivative work based on this package.
3. RESTRICTIONS
You may not rent, lease, license, disclose or give this package to any person or organization, or use the information contained in this
package, for the benefit of any third party or for any purpose other than as specified above unless such use is with the prior written
permission of EPRI.You agree to take all reasonable steps to prevent unauthorized disclosure or use of this package.Except as specified
above, this agreement does not grant you any right to patents, copyrights, trade secrets, trade names, trademarks or any other
intellectual property, rights or licenses in respect of this package.
4.TERM AND TERMINATION
This license and this agreement are effective until terminated.You may terminate them at any time by destroying this package. EPRI has
the right to terminate the license and this agreement immediately if you fail to comply with any term or condition of this agreement.
Upon any termination you may destroy this package, but all obligations of nondisclosure will remain in effect.
5. DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES
NEITHER EPRI,ANY MEMBER OF EPRI,ANY COSPONSOR, NOR ANY PERSON OR ORGANIZATION ACTING ON BEHALF
OF ANY OF THEM:
(A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I) WITH RESPECT TO THE USE
OF ANY INFORMATION,APPARATUS, METHOD, PROCESS OR SIMILAR ITEM DISCLOSED IN THIS PACKAGE, INCLUDING
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR
INTERFERE WITH PRIVATELY OWNED RIGHTS, INCLUDING ANY PARTYS INTELLECTUAL PROPERTY, OR (III) THAT THIS
PACKAGE IS SUITABLE TO ANY PARTICULAR USERS CIRCUMSTANCE; OR
(B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDING ANY
CONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THIS PACKAGE OR ANY INFORMATION,APPARATUS,
METHOD, PROCESS OR SIMILAR ITEM DISCLOSED IN THIS PACKAGE.
6. EXPORT
The laws and regulations of the United States restrict the export and re-export of any portion of this package, and you agree not to
export or re-export this package or any related technical data in any form without the appropriate United States and foreign
government approvals.
7. CHOICE OF LAW
This agreement will be governed by the laws of the State of California as applied to transactions taking place entirely in California
between California residents.
8. INTEGRATION
You have read and understand this agreement, and acknowledge that it is the final, complete and exclusive agreement between you
and EPRI concerning its subject matter, superseding any prior related understanding or agreement. No waiver, variation or different
terms of this agreement will be enforceable against EPRI unless EPRI gives its prior written consent, signed by an officer of EPRI.
2001 Electric Power Research Institute (EPRI), Inc. All rights

reserved. Electric Power Research Institute and EPRI are registered
service marks of the Electric Power Research Institute, Inc.
EPRI. ELECTRIFY THE WORLD is a service mark of the Electric
Power Research Institute, Inc.
Printed on recycled paper in the United States of America
1006537
EPRI 3412 Hillview Avenue, Palo Alto, California 94304 PO Box 10412, Palo Alto, California 94303 USA
800.313.3774 650.855.2121 askepri@epri.com www.epri.com

Boiler Reliabiliti Optimization 2001

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Boiler Reliabiliti Optimization 2001

Uploaded by

Copyright:

Available Formats

EPRI Boiler Reliability Optimization

EPRI Boiler Reliability Optimization

Final Report, December 2001

EPRI Project Manager

DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES

EPRI Licensed Material

1 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1, 2 & 4 AT AMERICAN

EXECUTIVE SUMMARY........................................................................................... 1-1

INTRODUCTION ...................................................................................................... 1-2

SCOPE OF WORK ................................................................................................... 1-3

FINDINGS AND OBSERVATIONS............................................................................ 1-3

1.4.1 Phase I and II - Technical and Organisational Assessment.................................. 1-3

EPRI Licensed Material

EPRI Licensed Material

2 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 2 BOILER

EPRI Licensed Material

3 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1 AT DETROIT

EPRI Licensed Material

4 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1- ROOT CAUSE

38,413 Mwh.......................................................... 4-7

Incident Description................................................................................................... 4-7

EPRI Licensed Material

4.4.10 Unit 4 HP Exciter Field Failure 33,500 Mwh .....................................................4-11

EPRI Licensed Material

4.16 Appendix 2 ...............................................................................................................4-40

EPRI Licensed Material

EPRI Licensed Material

EPRI Licensed Material

EPRI Licensed Material

Phase I - Technical and Organisational Assessments

Phase II - Development of a Boiler failure defence plan

Phase III - Implementation of a failure defence plan

Phase IV - Continuous Improvement and Process automation

EPRI Licensed Material

This consisted of the following activities:

FINDINGS AND OBSERVATIONS

1.4.1 Phase I and II - Technical and Organisational Assessment

EPRI Licensed Material

Work Management System

EPRI Licensed Material

W ork Identific ation

W ork M anagem ent S y s tem s

B enc hm ark ing

Inform ation Integration Tools

Com m unic ation Inter

Continuous Im provem ent

1.4.1.1 Work Process

EPRI Licensed Material

Work Flow Process

EPRI Licensed Material

Work Mgmt System

CBM Data Collection &

EPRI Licensed Material

Production Departments. Maintenance Management is then in a position to determine the best

The Boiler Specialist owns the Boiler System PM Basis.

EPRI Licensed Material

Corrective Maintenance Repair tasks (CM) - tasks as a result of loss-of-performance, component

Emergency maintenance work is discussed between operations and maintenance leaders,

EPRI Licensed Material

Planners prepare work packages for critical work orders.

EPRI Licensed Material

This is the final activity and includes work close-out.

Limited post maintenance testing is done.

2. Compile a management directive identifying the minimum acceptable standards to be