®

IBM Software Group

Diagnosing WebSphere Application

Server Hangs on AIX

Ricky Marley,
Advisory Software Engineer,
WebSphere Application Server L2 Support Team Lead

WebSphere® Support Technical Exchange

 IBM Software Group

Agenda
ƒ Background/Overview
ƒ What Data Should be Collected to Diagnose the
Problem
ƒ Javacores
- Basics
- Thread Analysis
- Lock Analysis
- Understanding Javacores from a WebSphere
AppServer
ƒ Summary
ƒ Questions?

WebSphere® Support Technical Exchange 2

 IBM Software Group

What is a Hang or Degradation in Performance?

ƒ Hang is a condition where the Java Virtual Machine (JVM)
become becomes unresponsive for client requests

ƒ When a client complains about hang, first thing to

understand is “What type of requests to WebSphere are
unresponsive?”

ƒ Starting V5.1.1, the below message will be written into

SystemOut file when a hung thread is detected:

[7/15/04 15:03:11:502 EDT] 3c3b4e37 ThreadMonitor W WSVR0605W: Thread

"Servlet.Engine.Transports : 0" (37c18e37) has been active for 680,839 milliseconds
and may be hung. There are 1 threads in total in the server that may be hung.

WebSphere® Support Technical Exchange 3

 IBM Software Group

What Can Cause a Hang?

ƒ Circular dependency in application code causing a
deadlock in JVM
ƒ Bottleneck caused by:
Improper tuning of the webserver, Web Container
(JVM), or database
Synchronization of Java code in the JVM
Web Container waiting for a response from an external
resource
Limitation of resources
ƒ CPU
ƒ Memory (typically Java heap)

WebSphere® Support Technical Exchange 4

 IBM Software Group

Threadpools and Their Role in the Appserver

ƒ Within the WebSphere Application Server process, there are various
types of thread pools that serve client requests:
` WebContainer Thread pool
` Object Request Broker (ORB) Thread pool
` Data Replication Service (DRS) Thread pool
` Java Message Service (JMS) Thread pool
` Alarm Thread pool that will be reused for various purposes
` SOAP Connector Thread pool
` Application defined Thread pool

ƒ The following link illustrates the various states in which you may find a
WebContainer thread:
http://www-1.ibm.com/support/docview.wss?uid=swg21137491

WebSphere® Support Technical Exchange 5

 IBM Software Group

Data Needed to Debug an AppServer Hang

Issue
ƒ Application logs alone cannot be used to debug this type of problem.
ƒ Documentation that needs to be collected at the time of the problem
can be found by clicking on the link for “Hangs / Performance
Degradation” at the following website:
http://www-1.ibm.com/support/docview.wss?uid=swg21145599
ƒ These documents include instructions for gathering:
Javacores
• Will be found in the AppServer Working Directory
WebSphere 5.X – /usr/WebSphere/AppServer
WebSphere 6.X - /usr/WebSphere/AppServer/profiles/server1
netstat
CPU utilization - tprof
Memory utilization – verbose GC data

WebSphere® Support Technical Exchange 6

 IBM Software Group

Introduction to Javacores – The Basics

A javacore , also known as a Java™ dump, Java™
thread dump or a thread dump, is a file that
contains a Large Amount of useful information
about the JVM. The data presented in the
following sections:
ƒ All of the threads that run on a Java™ Virtual
Machine (JVM).
ƒ All of the monitors on a JVM.
ƒ Some useful information about the system that
the JVM runs under.

WebSphere® Support Technical Exchange 7

 IBM Software Group

Introduction to Javacores – Dump Routines

The following chart presents the logical subcomponents
in which a Javacore presents information from the JVM.
To debug a performance degradation, focus should be
placed in the LK and XM subcomponents.
Title
XHPI Subcomponent
CI Subcomponent
Tags
LK Subcomponent
XM Subcomponent
CL Subcomponent

WebSphere® Support Technical Exchange 8

 IBM Software Group

Introduction to Javacores – Dump Routines

The following figure is an example of the “Title” dump
routine which presents the reason the JVM produced the
Javacore file. The TITLE subcomponent contains the date
and time of occurrence and the location within your file
system. You can use this information to determine if the
Java dump was created by a user or was system generated.
NULL ------------------------------------------------------------------------
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO signal 3 received
1TIDATETIME Date: 2006/08/29 at 14:47:04
1TIFILENAME Javacore filename: /usr/WebSphere/AppServer/javacore29534.1156877224.txt
NULL ------------------------------------------------------------------------

WebSphere® Support Technical Exchange 9

 IBM Software Group

Introduction to Javacores – Dump Routines

The following figure is an example of the “XHPI
subcomponent” dump routine. This section will identify
the operating environment, memory information, user
limits, and other operating system related details.
NULL ------------------------------------------------------------------------
0SECTION XHPI subcomponent dump routine
NULL ==============================
1XHTIME Tue Aug 29 14:47:04 2006
1XHSIGRECV SIGQUIT received at 0x0 in <unknown>.
1XHFULLVERSION J2RE 1.4.2 IBM AIX build ca1420-20040626
…
1XHOPENV Operating Environment
NULL ---------------------
2XHHOSTNAME Host : aixwas3.rtp.raleigh.ibm.com:9.42.115.17
2XHOSLEVEL OS Level : AIX 5.3.0.0
…
1XHENVVARS Environment Variables
NULL ---------------------
2XHENVVAR MANPATH=/usr/dt/man:/usr/share/man
….

WebSphere® Support Technical Exchange 10

 IBM Software Group

Introduction to Javacores – Dump routines

The following figure is an example of the “CI
subcomponent” dump routine. Use this section to identify
the JVM build, the class path that the JVM uses, the JVM
system property variables, and other system information
related to the JVM.
NULL ----------------------------------------------------------------
0SECTION CI subcomponent dump routine
NULL ============================
1CIJAVAVERSION J2RE 1.4.2 IBM AIX build ca1420-20040626
1CIRUNNINGAS Running as a standalone JVM
1CICMDLINE /usr/WebSphere/AppServer/java/bin/java -Djava.net.preferIPv4Stac
1CIJAVAHOMEDIR Java Home Dir: /usr/WebSphere/AppServer/java/bin/../jre
1CIJAVADLLDIR Java DLL Dir: /usr/WebSphere/AppServer/java/bin/../jre/bin
1CISYSCP Sys Classpath: /usr/WebSphere/AppServer/java/bin/../jre/lib/co
1CIUSERARGS UserArgs:
2CIUSERARG vfprintf 0x30000EF4
2CIUSERARG -Djava.net.preferIPv4Stack=true
2CIUSERARG -Dwas.status.socket=35676
2CIUSERARG -Xbootclasspath/p:/usr/WebSphere/AppServer/java/jre/li
2CIUSERARG -Xms50m
2CIUSERARG -Xmx256m

WebSphere® Support Technical Exchange 11

 IBM Software Group

Introduction to Javacores – Dump Routines

The following figure is an example of the “LK
subcomponent” dump routine. This Component provides
information for all of the monitors in the JVM. You can use
this section to analyze how resources are being used by
various threads.
0SECTION LK subcomponent dump routine
NULL
============================
NULL
1LKPOOLINFO Monitor pool info:
2LKPOOLINIT Initial monitor count: 32
2LKPOOLEXPNUM Minimum number of free monitors before expansion: 5
2LKPOOLEXPBY Pool will next be expanded by: 182
2LKPOOLTOTAL Current total number of monitors: 364
2LKPOOLFREE Current number of free monitors: 163
NULL
1LKMONPOOLDUMP Monitor Pool Dump (flat & inflated object-monitors):
2LKMONINUSE sys_mon_t:0x44F79558 infl_mon_t: 0x00000000:
3LKMONOBJECT java.lang.Object@359F1130/359F1138: Flat locked by thread ident 0x26
3LKNOTIFYQ Waiting to be notified:
3LKWAITNOTIFY "Servlet.Engine.Transports : 0" (0x450914A0)

WebSphere® Support Technical Exchange 12

 IBM Software Group

Introduction to Javacores – Dump Routines

The following figure is an example of the “XM subcomponent”
dump routine. This section provides a detailed list of all the
threads and their states. It also provides the current stack
trace for each thread listed. You can use the stack trace to
relate the current activity of thread with the source code.
0SECTION XM subcomponent dump routine
NULL ============================
NULL 1XMCURTHDINFO Current Thread Details
NULL ----------------------
3XMTHREADINFO "Signal dispatcher" (TID:0x300FB960, sys_thread_t:0x412CDAA0, state:R
3XHNATIVESTACK Native StackNULL ------------
3XHSTACKLINEERR unavailable - stack address not valid1
XMTHDINFO All Thread Details
NULL ------------------
2XMFULLTHDDUMP Full thread dump Classic VM (J2RE 1.4.2 IBM AIX build ca1420-20040626
3XMTHREADINFO "Servlet.Engine.Transports : 3" (TID:0x30285098, sys_thread_t:0x4660CD20
4XESTACKTRACE at TE02Servlet.doLock02(TE02Servlet.java:77)
4XESTACKTRACE at TE02Servlet.doPost(TE02Servlet.java:36)
4XESTACKTRACE at TE02Servlet.doGet(TE02Servlet.java:18)
4XESTACKTRACE at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
4XESTACKTRACE at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
. . .

WebSphere® Support Technical Exchange 13

 IBM Software Group

Introduction to javacores – Monitor analysis

The following table shows how the LK Dump
routine is broken down into five subsections:
Monitor Pool Info Provides basic monitor information, such as, the current total number of monitors,
and so on.

Monitor Pool Dump Lists the monitors that exist in the JVM at the time of the Java dump along with the
threads waiting for that particular monitor and the owner of each monitor. Monitors
can also be owned or not owned which will be indicated in the file. You can use this
section to identify any problems, such as, too many threads waiting on a monitor.

JVM System Similar to the monitor pool dump except that instead of listing any monitor, it lists all
Monitor Dump of the system monitors on the JVM for which the Java dump was captured.

Java Object The same as the monitor pool dump except that additional JVM internal information is
Monitor Dump provided.

Thread Identifiers Provides an association between the XM dump routine and the monitor pool dump.
You can use this section as a pointer to the location of the problem identified in the
monitor pool dump.

WebSphere® Support Technical Exchange 14

 IBM Software Group

Introduction to javacores – Thread analysis

Thread States

State Name Description

R Runnable Thread that has the ability to run or is

running.
CW Conditional Thread waiting on a condition variable.
Wait Typically threads in this state are waiting
for a certain condition to occur, for
example, a thread waiting for a resource
to become available.
MW Monitor Thread waiting on a monitor lock
Wait
S Suspended Thread suspended.

WebSphere® Support Technical Exchange 15

 IBM Software Group

Analyzing Javacores
ƒ There are many ways to approach analyzing the Javacores to
determine the cause of a hang or performance degradation.
It is suggested to use the following methodology in the order listed:
 Is there a deadlock?
 Is there a monitor that has a long list of threads waiting for it?
 What are the Servlet.Engine.Transport threads doing?
A. Are they idle waiting for requests?
B. Are they busy and many of them have the same or similar
stacks?
C. Are some or most of the threads busy and they all appear to
be doing something different?
ƒ Javacores can be analyzed manually or by using the ThreadAnalyzer
tool
(http://www-128.ibm.com/developerworks/websphere/downloads/thread_analyzer.html)

WebSphere® Support Technical Exchange 16

 IBM Software Group

Analyzing javacores - Deadlock

The Javacore will report a deadlock as follows:

1LKDEADLOCK Deadlock detected !!!
NULL ---------------------
NULL
2LKDEADLOCKTHR Thread "Servlet.Engine.Transports : 3" (0x4660CD20)
3LKDEADLOCKWTR is waiting for:
4LKDEADLOCKMON sys_mon_t:0x4532E7C8 infl_mon_t: 0x45170CC0:
4LKDEADLOCKOBJ java.lang.Object@359F1120/395F1128:
3LKDEADLOCKOWN which is owned by:
2LKDEADLOCKTHR Thread "Servlet.Engine.Transports : 0" (0x450914A0)
3LKDEADLOCKWTR which is waiting for:
4LKDEADLOCKMON sys_mon_t:0x44F79558 infl_mon_t: 0x00000000:
4LKDEADLOCKOBJ java.lang.Object@359F1130/395F1138:
3LKDEADLOCKOWN which is owned by:
2LKDEADLOCKTHR Thread "Servlet.Engine.Transports : 3" (0x4660CD20)

WebSphere® Support Technical Exchange 17

 IBM Software Group

Analyzing Javacores – Threads waiting on Monitor

ƒ The following output from a Javacore shows several
Servlet.Engine.Transport threads waiting on a monitor
named java.lang.Object@359F1140/359F1148
ƒ This monitor is owned by thread identifier 0x25

2LKMONINUSE sys_mon_t:0x44F75D48 infl_mon_t: 0x00000000:

3LKMONOBJECT java.lang.Object@359F1140/359F1148: Flat locked by thread
ident 0x25, entry count 1
3LKNOTIFYQ Waiting to be notified:
3LKWAITNOTIFY "Servlet.Engine.Transports : 1" (0x450EBAA0)
3LKWAITNOTIFY "Servlet.Engine.Transports : 0" (0x450914A0)
3LKWAITNOTIFY "Servlet.Engine.Transports : 3" (0x4660CD20)
3LKWAITNOTIFY "Servlet.Engine.Transports : 4" (0x4670CA14)
3LKWAITNOTIFY "Servlet.Engine.Transports : 5" (0x4760CF50)

WebSphere® Support Technical Exchange 18

 IBM Software Group

Analyzing Javacores – Threads waiting on Monitor

Examine the “Thread identifiers” section of the LK
subcomponent dump to determine the name of the thread
associated with “thread ident 0x25.” In this case, thread
ident 0x25 is “Servlet.Engine.Transports : 2”

1LKFLATMONDUMP Thread identifiers (as used in flat monitors):

2LKFLATMON ident 0x26 "Servlet.Engine.Transports : 3" (0x4660CD20)
2LKFLATMON ident 0x25 "Servlet.Engine.Transports : 2" (0x464900A0)
2LKFLATMON ident 0x14 "Servlet.Engine.Transports : 1" (0x450EBAA0)
2LKFLATMON ident 0x13 "Servlet.Engine.Transports : 0" (0x450914A0)

WebSphere® Support Technical Exchange 19

 IBM Software Group

Analyzing Javacores – Threads waiting on Monitor

“Servlet.Engine.Transports : 2” is found in the XM
component section of the Javacore. The Java stack trace
for thread is as follows:
3XMTHREADINFO "Servlet.Engine.Transports : 2" (TID:0x3018A658, sys_thread_t:0x464900A0,
state:CW, native ID:0x2438) prio=5
4XESTACKTRACE at java.lang.Thread.sleep(Native Method)
4XESTACKTRACE at TE02Servlet.doLock01(TE02Servlet.java:64)
4XESTACKTRACE at TE02Servlet.doPost(TE02Servlet.java:33)
4XESTACKTRACE at TE02Servlet.doGet(TE02Servlet.java:18)
4XESTACKTRACE at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
4XESTACKTRACE at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

The TE02Servlet has captured the lock and has gone to sleep.
Until this thread releases the monitor, all threads waiting for this monitor will be
blocked
Review subsequent Javacores and monitor the state of this thread

WebSphere® Support Technical Exchange 20

 IBM Software Group

Analyzing Javacores – Servlet Engine Transport

Threads
A. Threads are idle waiting for requests
ƒ Idle threads could indicate that requests are not making it into the JVM.
At this point, you need to determine if any HTTP clients have been able
to establish connections to the webserver/plugin or WebSphere
transport.
Is the transport thread present in the JVM?
Are there any established connections to the webserver or
application server?
Review the webserver’s plugin log for potential errors connecting to
the application server

WebSphere® Support Technical Exchange 21

 IBM Software Group

Analyzing Javacores – Servlet Engine Transport

Threads
B. Threads are busy and many of them have the
same or similar stacks
1. Most of the threads are Runnable (State: R) and the top of their stack
is java.net.SocketInputStream.socketRead

This indicates that there is not a bottleneck regarding synchronized

java resources. If the threads have similar stacks and Runnable, they
are most likely in socketRead waiting for a response from a remote
server.
 What does the stack of the thread look like?
 What is the thread waiting on (LDAP, Oracle, HttpURLConnection,
etc)?

WebSphere® Support Technical Exchange 22

 IBM Software Group

Analyzing Javacores – Servlet Engine Transport

Threads
B. Threads are busy and many of them have the same or
similar stacks
2. Most of the threads are in monitor wait or conditional wait (State: MW
or CW) indicating a potential bottleneck in synchronized code
 What does the stack look like for the threads that are in MW or CW?
 What thread holds the monitor that is blocking these threads? This
thread should be reviewed in each thread dump.
– Does the stack of this thread remain the same? If so, none of
the threads that are waiting on that monitor have been able to
continue processing.
– Does the stack of this thread change? If enough threads are
attempting to pass through the code that is synchronized and
the thread that holds the monitor takes a relatively long time to
release it, this could cause a hang or performance degradation

WebSphere® Support Technical Exchange 23

 IBM Software Group

Analyzing Javacores – Servlet Engine Transport

Threads
C. Threads are not idle, some or most of the
available threads are in use, and they all appear
to be doing something different. This can indicate
a resource issue.

ƒ Is CPU utilization high? High CPU will cause threads to contend for
CPU cycles
ƒ What does memory utilization look like?
ƒ What does VerboseGC data look like?
When GC runs, all other threads are suspended.

WebSphere® Support Technical Exchange 24

 IBM Software Group

Summary
ƒ Identifying when a JVM is encountering a
performance degradation

ƒ Collecting the documentation required to debug the

problem

ƒ Locating the data collected

ƒ Introduction to the content of javacores

ƒ Analyzing javacores for a WebSphere Application

Server JVM

WebSphere® Support Technical Exchange 25

 IBM Software Group

Additional WebSphere Product Resources

ƒ Discover the latest trends in WebSphere Technology and implementation,

participate in technically-focused briefings, webcasts and podcasts at:
www.ibm.com/developerworks/websphere/community/
ƒ Learn about other upcoming webcasts, conferences and events:
www.ibm.com/software/websphere/events_1.html
ƒ Join the Global WebSphere User Group Community: www.websphere.org
ƒ Access key product show-me demos and tutorials by visiting IBM Education
Assistant: ibm.com/software/info/education/assistant
ƒ Learn about the Electronic Service Request (ESR) tool for submitting
problems electronically:
www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf
.html
ƒ Sign up to receive weekly technical My support emails:
www.ibm.com/software/support/einfo.html

WebSphere® Support Technical Exchange 26

 IBM Software Group

Questions and Answers

WebSphere® Support Technical Exchange 27