Exadata Storage Server gives a warning of unsupported firmware during boot if mixed Flash F40

firmware is installed [ID 1529290.1]

Applies to:

SPARC SuperCluster T4-4 Full Rack - Version All Versions and later
SPARC SuperCluster T4-4 - Version All Versions and later
Exadata X3-2 Hardware - Version All Versions and later
Oracle Exadata Storage Server Software - Version 11.2.3.2.0 to 11.2.3.2.0 [Release 11.2]
Exadata X3-2 Quarter Rack - Version All Versions and later
Information in this document applies to any platform.
Exadata Storage Server gives a warning of unsupported firmware during boot if Flash F40 cards with
different firmware versions are detected.

Symptoms

Exadata Storage Server with image 11.2.3.2.0 gives a warning of unsupported firmware during boot if Flash
F40 cards with different firmware versions are detected, for example after a F40 card replacement:

[root@dbm1cel01 ~]# /opt/oracle.SupportTools/CheckHWnFWProfile -c strict

[WARNING] The hardware and firmware are not supported. See details below

[PCISlot:CardId:MinCardFw:CardModel:DOMId:OSDevice:MinDOMfw:DOMCount]

Requires:

All All 06.05.10.00 ELP-4x100-4d-n All All TI35 All_of_16

Found:

5 1 06.05.10.00 ELP-4x100-4d-n 1 /dev/sdn TI35 1_of_16

5 1 06.05.10.00 ELP-4x100-4d-n 2 /dev/sdo TI35 2_of_16

5 1 06.05.10.00 ELP-4x100-4d-n 3 /dev/sdp TI35 3_of_16

5 1 06.05.10.00 ELP-4x100-4d-n 4 /dev/sdq TI35 4_of_16

4 2 08.05.01.00 ELP-4x100-4d-n 1 /dev/sdr UI39 5_of_16

4 2 08.05.01.00 ELP-4x100-4d-n 2 /dev/sds UI39 6_of_16

4 2 08.05.01.00 ELP-4x100-4d-n 3 /dev/sdt UI39 7_of_16

4 2 08.05.01.00 ELP-4x100-4d-n 4 /dev/sdu UI39 8_of_16

1 3 08.05.01.00 ELP-4x100-4d-n 1 /dev/sdv UI39 9_of_16

1 3 08.05.01.00 ELP-4x100-4d-n 2 /dev/sdw UI39 10_of_16

1 3 08.05.01.00 ELP-4x100-4d-n 3 /dev/sdx UI39 11_of_16

1 3 08.05.01.00 ELP-4x100-4d-n 4 /dev/sdy UI39 12_of_16

2 4 08.05.01.00 ELP-4x100-4d-n 1 /dev/sdz UI39 13_of_16

2 4 08.05.01.00 ELP-4x100-4d-n 2 /dev/sdaa UI39 14_of_16

2 4 08.05.01.00 ELP-4x100-4d-n 3 /dev/sdab UI39 15_of_16

2 4 08.05.01.00 ELP-4x100-4d-n 4 /dev/sdac UI39 16_of_16

[WARNING] The hardware and firmware are not supported. See details above

Cause

This issue is caused because Flash F40 firmware was updated in the factory on the Exadata prior to
shipping, with firmware 08.05.01.00 and FMOD firmware UI39 (SW1.1), and a F40 card was installed with
older firmware 06.05.10.00 and FMOD firmware TI35 (SW1.0). Exadata image 11.2.3.2.0 supports running
the later firmware but does not have the firmware files in it to upgrade the older card. The warning is
generated due to having different firmware on different F40 cards in the same system.

Solution

Update Exadata Storage Server image to v11.2.3.2.1 or later that includes Flash F40 card firmware
08.05.01.00 and FMOD firmware UI39 (SW1.1). Refer to MOS Note 888828.1 for details.

If this is not possible at this time, the following solution may be used:

The Flash F40 card with older firmware should be upgraded to SW1.1 manually using the firmware and
'ddcli' utility contained in patch 16005846. This should be done on all down-rev Flash F40 cards so that the
firmware matches on all 4 Flash F40 cards in the Exadata Storage Server. The cell services will need to be
offlined first to ensure the system is not doing I/O to the Flash F40 card being upgraded.

1. Download patch 16005846.

2. Copy the patch to the Exadata Storage Server, make a directory "p16005846" and unzip the patch file
into this directory.

[root@dbm1cel01 ~]# mkdir p16005846

[root@dbm1cel01 ~]# cd p16005846
[root@dbm1cel01 p16005846]# unzip ../p16005846_11_Generic.zip

3. As 'root', change permissions to the patch "Linux/Utilities/DDCLI_linux_x86_64/ddcli" utility to

allow execute:

[root@dbm1cel01 p16005846]# chmod +x Linux/Utilities/DDCLI_linux_x86_64/ddcli

4. Offline the Exadata Storage Server from ASM prior to flash updating the Flash F40 card.

a. By default, ASM drops a disk shortly after it is taken offline; however, you can set the
DISK_REPAIR_TIME attribute to prevent this operation by specifying a time interval to
repair the disk and bring it back online. The default DISK_REPAIR_TIME attribute value of
3.6h should be adequate for this procedure.

i. To check repair times for all mounted disk groups - log into the ASM instance on the
DB node and perform the following query:
 SQL> select dg.name,a.value from v$asm_diskgroup
dg, v$asm_attribute a where dg.group_number=a.group_number and
a.name='disk_repair_time';

ii. The disks should be offline for no more than 1 hour for this procedure, so if you are
not set for the 3.6h default value and are set for less than 1 hour, then adjust the
parameter by issuing the command below as an example:

SQL> ALTER DISKGROUP DATA SET ATTRIBUTE

'DISK_REPAIR_TIME'='1.0H';

b. Next you will need to check if ASM will be OK if the grid disks go OFFLINE. The following
command should return 'Yes' for the grid disks being listed:

[root@dbm1cel01 p16005846]# cellcli -e list griddisk attributes

name,asmmodestatus,asmdeactivationoutcome

c. If one or more disks return asmdeactivationoutcome='No', you should wait for some time and
repeat step #2. Once all disks return return asmdeactivationoutcome='Yes', you can proceed
with taking the griddisk offline in the next step.

Note: Taking the storage server offline when one or more grid disks return
asmdeactivationoutcome='No' will cause Oracle ASM to dismount the affected disk group,
causing the databases to shut down abruptly.

d. Run cellcli command to Inactivate all grid disks on the cell:

[root@dbm1cel01 p16005846]# cellcli -e alter griddisk all inactive

* Please note - This action could take 10 minutes or longer depending on activity. It is very
important to make sure you were able to offline all the disks successfully before shutting
down the cell services. Inactivating the grid disks will automatically OFFLINE the disks in
the ASM instance.

e. Confirm that the griddisks are now offline by performing the following actions:

i. Execute the command below and the output should show asmmodestatus=UNUSED
and asmdeactivationoutcome=Yes for all griddisks once the disks are offline in ASM.
Only then is it safe to proceed:

[root@dbm1cel01 p16005846]# cellcli -e list griddisk attributes

name,asmmodestatus,asmdeactivationoutcome

( there has also been a reported case of asmmodestatus= OFFLINE: Means Oracle
ASM has taken this grid disk offline. This status is also fine and can proceed with
remaining instructions)

ii. List the griddisks to confirm all now show offline:

[root@dbm1cel01 p16005846]# cellcli -e list griddisk

f. Shutdown the cell services:

[root@dbm1cel01 p16005846]# cellcli -e alter cell shutdown services cellsrv

5. Identify which controller ID's need updating. Exadata Storage Server's have 4 Flash F40 cards in
them. In general the cards are mapped as follows:

o controller id 1 - PCIe Slot 5

o controller id 2 - PCIe Slot 4

o controller id 3 - PCIe Slot 1

o controller id 4 - PCIe Slot 2

[root@dbm1cel01 p16005846]# Linux/Utilities/DDCLI_linux_x86_64/ddcli -listall

****************************************************************************
LSI Corporation WarpDrive Management Utility
Version 107.00.00.05 (2012.09.07)
Copyright (c) 2011 LSI Corporation. All Rights Reserved.
****************************************************************************

ID WarpDrive Package Version PCI Address

-- --------- --------------- -----------
1 ELP-4x100-4d-n 06.05.10.00 00:20:00:00
2 ELP-4x100-4d-n 08.05.01.00 00:30:00:00
3 ELP-4x100-4d-n 08.05.01.00 00:90:00:00
4 ELP-4x100-4d-n 08.05.01.00 00:b0:00:00

LSI WarpDrive Management Utility: Execution completed successfully.

[root@dbm1cel01 p16005846]#

Update the controller ID which has firmware 06.05.10.00 using the format "ddcli -c <ID>
-updatepkg <flash binary package file>". This command supports the upgrade of only the firmware
package. If the current firmware package version on the selected card is higher than the specified
firmware package version, the command returns an error.
In the example above, this is controller ID 1:

[root@dbm1cel01 p16005846]# Linux/Utilities/DDCLI_linux_x86_64/ddcli -c 1 -updatepkg

08.05.01.00/ELP-4x100-4d-n_08.05.01.00.bin

****************************************************************************
LSI Corporation WarpDrive Management Utility
Version 107.00.00.05 (2012.09.07)
Copyright (c) 2011 LSI Corporation. All Rights Reserved.
****************************************************************************
LSI WarpDrive Management Utility: Package Type: 0x7, PCI SubSystem ID: 0x581
LSI WarpDrive Management Utility: Please wait. Package update is in progress..

HBA Legacy BIOS Download...

Attempting to Flash BIOS Image...

Flash BIOS Image Successful.

 . Flashing SSD Firmware....

.
HBA Firmware Download...

NVDATA Versions Compatible.

Beginning Firmware Download...

Firmware Download Successful.

Verifying Download...

Firmware Flash Successful!

Resetting Adapter...

Adapter Successfully Reset.

Flash Firmware Image Successful.

LSI WarpDrive Management Utility: WarpDrive package updated successfully.

LSI WarpDrive Management Utility: Execution completed successfully.

If a controller firmware update fails, the -updatepkg command terminates.

If a firmware download fails on any of the card components, the process terminates.

If the firmware update fails, open a SR or reengage the SR owner if an SR is already opened.

. Repeat step 6 for each downrev Flash F40 card identified in step 5.

. Reboot the Exadata Storage Server.

[root@dbm1cel01 p16005846]# shutdown -r now

The firmware profile check should now pass during boot without warning, returning SUCCESS on
the console. This can be confirmed manually, when the system comes back online using the
following command:

[root@dbm1cel01 ~]# /opt/oracle.SupportTools/CheckHWnFWProfile -c strict

[SUCCESS] The hardware and firmware profile matches one of the supported profiles

Once the Exadata Storage Server comes back online the cell services will start up automatically,
however you will need to reactive the griddisks:

[root@dbm1cel01 ~]# cellcli -e alter griddisk all active

. Verify all disks show 'active':

[root@dbm1cel01 ~]# cellcli -e list griddisk

. Verify grid disk status:

Verify all grid disks have been successfully put online using the following command:

[root@dbm1cel01 ~]# cellcli -e list griddisk attributes name, asmmodestatus

Wait until asmmodestatus is ONLINE for all grid disks. Each disk will go to a 'SYNCING' state first
then 'ONLINE'. The following is an example of the output:

DATA_CD_00_dm01cel01 ONLINE
DATA_CD_01_dm01cel01 SYNCING
DATA_CD_02_dm01cel01 OFFLINE
DATA_CD_03_dm01cel01 OFFLINE
DATA_CD_04_dm01cel01 OFFLINE
DATA_CD_05_dm01cel01 OFFLINE
DATA_CD_06_dm01cel01 OFFLINE
DATA_CD_07_dm01cel01 OFFLINE
DATA_CD_08_dm01cel01 OFFLINE
DATA_CD_09_dm01cel01 OFFLINE
DATA_CD_10_dm01cel01 OFFLINE
DATA_CD_11_dm01cel01 OFFLINE

Oracle ASM synchronization is only complete when all grid disks show asmmodestatus=ONLINE.

(Please note: this operation uses Fast Mirror Resync operation - which does not trigger an ASM
rebalance. The Resync operation restores only the extents that would have been written while the
disk was offline.)

This firmware should only be installed as necessary. If another Exadata Storage Server requires it,
then before repeating the procedure and taking the next Exadata Storage Server offline, Oracle ASM
synchronization must complete on the restarted Oracle Exadata Storage Server. If synchronization is
not complete, then the check performed on another storage server will fail. The following is an
example of the output:

CellCLI> list griddisk attributes name where asmdeactivationoutcome != 'Yes'

DATA_CD_00_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_01_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_02_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_03_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_04_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_05_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_06_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_07_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_08_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_09_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_10_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"
DATA_CD_11_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup"

. The patch files can be removed from the Exadata Storage Server.
.
REFERENCES:
Sun Flash Accelerator F40 PCIe Card Documentation Library contains Service instructions for
firmware upgrade in the User's Guide - http://docs.oracle.com/cd/E29748_01/index.html