MAINTENANCE DEPARTMENT

LESSON LEARNT FROM INCIDENT IG16-1105:

P-1107A WAS BURNT OUT AND
CAUSED HEATER H-1101 TO SHUTDOWN

20 June 2016

ISO 9001; ISO 14001; OHSAS 18001; ISO/IEC 17025

MAINTENANCE DEPARTMENT

Introduction

Finding and root causes

Countermeasures and lesson learnt

MAINTENANCE DEPARTMENT

Introduction

Before incident: CDU was normal operation with 105%

capacity and Jet A1 mode
At 08:35, 29/04/2016: PM-1107A was burnt and activated CB
of 08B-SW-4-1-1B (Incomer) that leading to stop motors
8:40 AM Heater was emergency stopped via Manual activate
UXHS-005-AA
At 11:00 AM,29/04/2016: restarted Heater H-1101 and starting
up CDU
At 7h15 PM, 29/04/2016: Switch all products to on-spec tanks
at 80% capacity

MAINTENANCE DEPARTMENT

Introduction

Temporary Loss Prevention Emergency Action done:

Switch products to crude recirculation header and run tank to tank.
Press button UXHS-005-AA to activate trip Heater including heater forced
draft fan B-1101A/B, pilot, fuel gas, fuel oil and off gas system to avoid
exploding heater because heater forced draft fan B-1101B was stopped but
pilot gas, fuel gas in service.
KTU was bypassed. Keep caustic recycle P-1401A/B and recycle water P1404A/B running.
All products (except LPG) were switched to re-circulation header and stop
hot feed for NHT and RFCC
At 8:53 AM, re-energized power for 8B-SW-4-1-Bus B after isolating
(Rack out MCC) of motor PM-1107A (winding was burnt).
Checked the power, control circuit and replaced the contactor and fuses
Replaced burnt out motor by spare one.

MAINTENANCE DEPARTMENT

Findings

Weather: dry, normal wind.

Operating procedure compliance: as per the operation log book, there is no
violation found
Before incident, P-1107A was running 75m3/h with allowable operating
region 28-128m3/h.
At the time, heater forced draft fan B-1101B was stopped due to loss of
power supply from bus B 400V in SS8B but 011FI067 was indicated about
55000Nm3/h (higher than trip flow rate about 34937Nm3/h) so pilot gas,
fuel gas, fuel oil and fuel gas system were not activated to trip. And it may
cause to explode heater H-1101.
Operation didnt received any information or alarm of bus bar B of 8BSW-4-1 lost power except alarms of motors stop status and related alarms.

MAINTENANCE DEPARTMENT

Findings
Operating procedure compliance: as per the operation log
book, there is no violation found.
No maintenance activity performed at the time of incident.
Maintenance history (as recorded in CMMS system):
Monthly monitoring and 3-month offline testing with WO
number 15-1431218 on 29th Dec 2015. Motor in normal
condition.
Preventive maintenance for LV motor PM-1107A with WO
number 16-1049940 on 29th Feb 2016. Motor in normal
condition.

MAINTENANCE DEPARTMENT

Findings

Similar failure history (as recorded in CMMS system):

PM-1118A burnt out on 29th March 2010.
PM-1516A burnt out on 07th July 2012.
PM-1521B burnt out on 07th September 2014.
PM-2101B burnt out on 12th February 2015.
PM-1110B burnt out on 22th February 2015.

MAINTENANCE DEPARTMENT

Findings
Event recorded in EMCS:
At 8:35:24 on 29th April 2016, PM-1107A was tripped with the
event on EMCS Lockout relay operated 86.
Motor specification:
Rated power: 75kW
Pole number: 2
Manufacturer type: FLSN 280S
Manufacturer: Leroy Somer
Protection: Ex n, IIB, T3

MAINTENANCE DEPARTMENT

Findings

Site check:
Incomer 08B-SW-4-1-1B tripped by Earth fault, undervoltage in
REF542 HMI.
Motor tripped by Undervoltage, earth fault and TOL in INSUM HMI.
A power fuse was blown at phase L1.

MAINTENANCE DEPARTMENT

Findings
Site check:
Contactor damaged (contacts were melted inside)
Check the motor after incident:
Insulation resistance: (500Vdc testing)
U-V : 0; V-W: 0; U-W: 0;
V-E : 0; W-E: 0; U-E: 0;
DC winding resistance:
U1-U2: 0.04; V1-V2: 0.2; W1-W2: 0.4
U-V: 0.6; V-W: 0.6; U-W: 0.7
Motors winding PM-1107A was burnt out.
There is no trace of motors water ingress.
Motor is operating at around 80% capacity (94A), under rated capacity.
Incomer 08B-SW-4-1-1B was tripped due to earth fault.
The pump P-1107A is in normal condition (checked and confirmed by ROT
team)

MAINTENANCE DEPARTMENT

Findings
Site check:
Motors winding PM-1107A was burnt out at NDE.

MAINTENANCE DEPARTMENT

Findings

Maintenance procedure/work steps:

Maintenance program for LV motor shall be as follow:
Yearly maintenance;
Re-greasing (depend on kind of bearing);
Monthly monitoring and 3-month offline testing;
Motor vibration is checked/monitored by Diagnostic team
(Rotating). Motor bearing replacement is recommended by
Diagnostic team;
Motor bearing temperature is checked/monitored by
Operation;
Overhaul for LV motor to be done depend on the equipment
history, this is condition based, not time based maintenance.

MAINTENANCE DEPARTMENT

Findings
Earth fault protection coordination 400V system:
With the existing design, the earth fault function for LV motor
is only applied for low earth fault current, high earth fault
current shall be protected by power fuses. And sometimes the
earth fault protection coordination between LV motor and
400V SWB incomer is not perfect. Some actions already done
to improve the earth fault protection coordination between LV
consumers and 400V SWB incomer.

MAINTENANCE DEPARTMENT

Findings
Earth fault protection coordination 400V system:

MAINTENANCE DEPARTMENT

Findings
Earth fault protection coordination 400V system:
Earth fault protection setting

Motor:
Low earth fault current: MCU 25A, 0,4s.
High earth fault current: Power fuse 315A.
400V switchboard incomer:
Relay Ref 542Plus:
CT ratio: 3200:1
Inst: 0,25 x CT sec (800A)
Time delay: 0,7s
Relay MK2200:
CT ratio: 1600:1
Inst: 0,51 x CT sec (816A)
Time delay: 0,95s

MAINTENANCE DEPARTMENT

Findings
Motor earth fault protection

MAINTENANCE DEPARTMENT

Findings
Motor earth fault protection

MAINTENANCE DEPARTMENT

Findings
Motor earth fault protection

MAINTENANCE DEPARTMENT

Findings
400V switchboard incomer earth fault protection

MAINTENANCE DEPARTMENT

Findings

MAINTENANCE DEPARTMENT

Findings
PM-1107A

MAINTENANCE DEPARTMENT

Findings
Earth fault protection coordination 400V system:
In 2014 (second Turnaround): implemented MOC-SUB-14-150 Carry out
Vendor recommendation to increase selectivity of earth fault protection for
400V switchboards in Refinery based on the Contract No. 453-2013/HDBDR-ENERSEV;
In 2015: On 31/3/2015 Electrical team already issued MOC-ELE-15-212
Change the MCC components to increase the selectivity of earth fault
protection for 400V switchboards in Refinery based on recommendations of
Contractor. On 30th December 2015, Electrical team already issued Service
request No. 453/DH-BDSC-E Request supply service to implement MOCELE-15-212 to increase reliability of earth protection for 400V
switchboards, this service will be applied for substation 1.
In 2016: Electrical team already issued Service request 328/DH-BDSC-E
dated 27/5/2016 Request supply service to implement MOC-ELE-15-212 to
increase reliability of earth protection for 400V switchboards, this service
will be applied for substations 4/8A/8B.

MAINTENANCE DEPARTMENT

Findings

MAINTENANCE DEPARTMENT

5-WHY
I. Incomer B of 8B-SW-4-1-1B activated
by earth fault
No

Question

Answer

Why heater shutdown?

Due to activation of logic UXHS005-AA

Why logic UXHS-005-AA

activated?

By the operator command

Why operators command

for s/d heater?

Because safety condition was

not meet (low air flow)

Why safety operating

condition was not meet?

Though BM-1101B (Fan) was

stopped, due to natural draft
through the open inlet guide
vane(FV-502B) and stack
damper, combustion air flow to
heater was higher than trip
value (34937.8 Nm3/h)

Why BM-1101B stopped?

Due to activation of earth fault

protection of incomer B 8B-SW-

MAINTENANCE DEPARTMENT

5-WHY
I. Incomer B of 8B-SW-4-1-1B activated
by earth fault
No

Question

Answer

Why incomer B of 8B-SW-41-1B activated by earth

fault protection?

Due to motor protection layer

did not activated as per design

Why motor feeder

protection not activated as
per design?

Due to the power fuses did not

activate as per the design

Why power fuses did not

activate as per the design

Because earth fault protection

coordination between
downstream (power fuses/MCU
for LV motors) and upstream
(relay REF 542 Plus for 400V
SWB incomer) is improper

MAINTENANCE DEPARTMENT

5-WHY
II. Motor PM-1107A burnt out
N
o

Question

Answer

Why pump P1107A stopped?

Because motor PM-1107A tripped

Why motor PM1107A tripped

Due to motor winding PM-1107A burnt out

Why motor PM1107A burnt out?

Motor winding of PM-1107A turn to turn

shorted.

Why motor
winding of PM1107A turn to
turn shorted?

Motor winding shorted turn to turn due to

defect of internal motor winding insulation
during operation

Why turn to turn

fault not
identified in
existing PM work?

In existing PM work, this problem could not be

detected clearly due to limitation of
technology/equipment. (Insulation resistance
testing can detect defect of insulation from
phase to earth and phase to phase but not turn

MAINTENANCE DEPARTMENT

Immediate causes
Failure to detect/measure
Motor stators winding was shorted turn to turn

MAINTENANCE DEPARTMENT

Technical root causes

Motor PM-1107A burnt:
Motor winding shorted turn to turn due to insulation defected internal
motor winding during operation, this problem could not detect by
insulation testing (Phase-Phase; Phase - Earth); turn to turn shorted to
cause to overheat excess heat will cause the enamel insulation on the
winding to break down and melt and internal short between the windings
will then do the rest as the current will go up further and result in more
heat and motor will burnt or just short out blowing the fuses/breaker.
Inadequate availability of tool/ equipment/machinery/device: caused
trip Incomer 8B-SW-4-1-1B by earth fault:
Improper design of 400V protection system: Earth fault protection
coordination between LV motors and 400V SWB incomer is not perfect
(refer to MOC-ELE-15-212 Change the MCC components to increase the
selectivity of earth fault protection for 400V switchboards in Refinery
based on recommendations of Contractor. ).

MAINTENANCE DEPARTMENT

Control Areas for Improvement Actions

MOC-ELE-15-212 Change the MCC components to increase the selectivity of

earth fault protection for 400V switchboards in Refinery should be
implemented as soon as possible to avoid the occurrence of similar incidents.
Need to enhance coordination between Electrical operation, Electrical
maintenance and Operation: make detail equipment list and attach all Low
Voltage switchboards and informed the available equipment to Operation in
case partial power loss in Low Voltage switchboard.
Study to apply for tripping fuel system if both motors of heater forced draft
fans are stopped:
Short term: develop logic of stopping fuel gas valve 011PV092 and fuel oil
valve 011PV118 if status of motors 011M069 (air blower B-1101B) and
011M076 (air blower B-1101A) are STOP
Long term: Develop logic of tripping pilot system if status of motors 011M069
(air blower B-1101B) and 011M076 (air blower B-1101A) are STOP ( modify
following boiler logic)

MAINTENANCE DEPARTMENT

Corrective and preventive actions

To apply motor circuit analyzer ALL test Pro IV for PM 3-month

offline testing
Issue Service request Request supply service to implement MOC-ELE-15212 to increase reliability of earth protection for 400V switchboards, this
service will be applied for substations 4/8A/8B.
Make detail equipment list and attach at all Low Voltage switchboards and
informed the available equipment to Operation in case partial power loss in
Low Voltage switchboard.
Study to apply for tripping fuel system if both motors of heater forced draft
fans are stopped
Modify check sheet to monitor weekly the temperature of motor at the DE
bearing temperature, Body temperature and terminal box temperature near
body

MAINTENANCE DEPARTMENT

Lesson learnt
Study to apply the new technology/methodology on
preventive/predictive maintenance for motor to improve the
reliability of motor.
Expedite the PR/modification for the approved MOC to
increase the selectivity of earth fault protection for 400V
switchboards, MOC-ELE-15-212 Change the MCC
components to increase the selectivity of earth fault protection
for 400V switchboards in Refinery;
Update the specification for Dung Quat Refinery Upgrading &
Expansion Project