Linux kernel hangs: BUG: soft

lockup and friends

Reasons and troubleshooting.
 Possible reasons for soft lockups
One of below:
1. Extraordinary load harmless.
2. Extraordinary temporary slowness of underlying hardware
(E.g. ESXi issues) harmless.
3. Race in the Linux kernel code. Dangerous - will never
recover. Races have greater chances to be triggered in (1) or
(2) cases or any during any other hardware/software issues.
This is because windows to trigger a race are very narrow in
normal circumstances. Any issue that makes that window
wider is greatly increasing the change the race to be
triggered.
 Kernel hangs
There are also plenty of other reasons besides
soft lookups why Linux could get into hang state.
E.g.
deadlock in any crucial section
Bugs in kernel memory allocator
Hardware issues
 Troubleshooting
If any issus in the kernel code is suspected crash dump should
be triggered for further RH analysis.
Five options to trigger crash dump:
1. Manual via SysRQ
2. Automatic kernel.softlockup_panic=1
3. Manual via NMI from ESXi shell. kernel.unknown_nmi_panic=1
4. Manual via VM snapshot from vcenter
5. Manual via VM suspend via vcenter
 SysRQ in VM environment
Prerequiste. VM console must be in text mode. E.g. init 3.
Prerequisite. vcenter HTTP interface should be used. VM
console should be opened in FULL SCREEN MODE
SysRQ commands are invoked by three buttons:
Alt + PrtSc + <KEY> combination.
Where KEY is actual SysRQ command. You can get the list of all
possible command by issuing combo Alt + PrtSc + h
Crash dump combo is: Alt + PrtSc + c
 Indication of a successuful kdump trigger
via SysRQ
In few minutes at the bottom of you screen you should see a % progress
indicator. Depending on the setting it could go over several phases.
filter: some pages (e.g. free pages)
compress
copying data last phase is the must for any setup. Observe increment
of the %.
 kernel.softlockup_panic=1 for VMs
Advantages over SysRQ
Not require manual intervention
Disadvantages
There are legitimate reasons for rare soft lockup events in
virtual environment (refer to p2) system eventually will
recover in these cases. Force kernel to panic in legitimate
cases is an overreaction. This is not safe even in a middle
term.
Possible solution
Turn kernel.softlockup_panic=1 temporarily until critical problem
is remediated. Turn it off afterwards.
kernel.unknown_nmi_panic=1 for VMs
A non-maskable interrupt (NMI) is a hardware interrupt that cannot be
ignored by standard interrupt masking techniques in the system.
Advantages over SysRQ
Works even in graphical mode.
Disadvantages
Require access to ESXi shell. UNIX SA lack this. We are depending on
Windows team presence during issues.
NMI is tricky to implement correctly from kernel side even on real
hardware. VMWare case creates additional complications.
kernel.unknown_nmi_panic=1 even in middle term might create
additional issues which overweight help from it.
Possible solution
Turn kernel.unknown_nmi_panic=1 temporarly untill we go life. Turn it off
when production phase will begin.
 Crash dump via NMI example
VM
[root@dca-eccs-sbx-db2 ~]# sysctl -w kernel.unknown_nmi_panic=1
kernel.unknown_nmi_panic = 1 Panic on NMI enabled.
[root@dca-eccs-sbx-db2 ~]#

ESXi shell
~ # esxcli vm process list
dca-eccs-sbx-db2
World ID: 42092
Process ID: 0
VMX Cartel ID: 42091
UUID: 42 39 4f 26 6b 4d 87 43-78 19 e5 92 db 79 da 28
Display Name: dca-eccs-sbx-db2
Config File: /vmfs/volumes/524c9c7d-dd35f261-4d42-0025b541a33f/dca-eccs-sbx-
db2/dca-eccs-sbx-db2.vmx

~ # /sbin/vmdumper 42092 nmi NMI to VM has been sent. Crash dump triggered
Sending NMI to guest...
~ #
Vmcore Via VM snapshot from vcenter

Advantages over SysRQ

Works even in graphical mode and in cases when kdump has failed
or SysRQ is not available due to misconfiguration (e.g. double
crashkernel boot option)
Disadvantages
Doesnt work for machine with RDMp LUNs. E.g. All current SAP
PROD environment
Dog slow for huge VMs up 3 hours in case of 256GiB memory.
Vmcore size is equal to the memory size RH case: 01164419
E.g. of snapshot creation via vcenter

Next step is contact Wu/Rao/Amnon D. And ask those person to

provide you .vmss or .vmsn corresponding to the snapshot you
created. Resulting snapshot file on the ESXi side has .vmsn suffix
 Vmcore Via VM suspend from vcenter
Advantages over SysRQ
Works even in graphical mode and in cases when kdump has failed or SysRQ is
not available due to misconfiguration (e.g. double crashkernel boot option)
Works even for RDMp machines
Disadvantages

Require access to ESXi shell. UNIX SA lack this. We are depending on Windows
team presence during issues.
Downtime is enormously long. Up to 4 hours Due to the nature of operation VM
need to be down during all next steps:
a) Suspend action (up to 3 hours)
b) Time required to contact Rao/Wu (Who is responsible in between)? Say 15
minutes
c) Time required to copy suspend image from ESXi to the Linux server. Up to one
hour.
Vmcore size is equal to the memory size RH case: 01164419
 E.g. vmcore Via VM suspend from vcenter

Next step is contact Wu/Rao/Amnon D. And ask those person to

provide you .vmss or .vmsn corresponding to the snapshot you
created. Resulting suspend file on the ESXi side has .vmss suffix
 .vms(s|n) to vmcore
Actual snapshot/suspend file name on VMFS folder on ESXi host could not
match the name you provided for a snapshot. This might be very confusing
be aware of that. E.g:
dca-eccs-sbx-db2-Snapshot5.vmsn is vmcore-20140727 from the snapshot page
dca-eccs-sbx-db2-66478b17.vmss is suspend file from the suspend page
Ask Windows Team/virtualization person
a) Open outdoing connection to 22 port on ESXi host
b) scp .vms(s|n) from ESXi shell to any Linux server E.g.
# scp dca-eccs-sbx-db2-Snapshot5.vmsn root@dca-unix-lab2:/var/crash/
or
# scp dca-eccs-sbx-db2-66478b17.vmss root@dca-unix-lab2:/var/crash/
c) Vmss2core on Linux server to convert to RH supported format
# vmss2core -N6 dca-eccs-sbx-db2-Snapshot5.vmsn
or
# vmss2core -N6 vmss2core -N6 dca-eccs-sbx-db2-Snapshot5.vmsn