<Insert Picture Here>

Thread Dump Analysis Muthukumar Subramaniam

Agenda

Thread Dump analysis

Thread Queues

weblogic.kernel.Default Worker threads that serve the external client requests. weblogic.kernel.system Internal system work likeRJVM heartbeats,Http state Dumps for JNDI updates in a cluster etc Weblogic.socket.Muxer- Defaults to 3 on Unix systems and 2 on Windows.Used for socket reads Weblogic.admin.rmi- Handle OA& M requests like deployment of application,Application poller etc Weblogic.admin.html- only on admin server to handle console requests. Core health monitor runtime health of the server JmsDispatcher, JMS.TimerTreePool, JMS.TimerClientPool -for jms

Thread States Life States

alive

This is a normal, running thread. Virtually all threads in the thread dump will be alive. The thread has been requested to start running by java.lang.Thread.start(), but the actual OS process has not yet started, or executed far enough to pass control to the JRockit JVM. It is extremely unlikely to see this value. A java.lang.Thread object that is created, but not has had start() executed, will not show up in the thread dump. This thread has finished its run() method and has also notified any threads joining on it, but it is still kept in the JVM internal thread structure for running threads. It is extremely unlikely to see this value. A thread that has been terminated for a time longer than a few milliseconds will not show up in the thread dump.

not started

terminated

Thread States Run States

This thread has tried to enter a synchronized block, but the lock was taken by another thread. This thread is blocked until the lock gets released.

blocked

blocked (on thin lock)

waiting
sleeping parked

This is the same state as blocked, but with the additional information that the lock in question is a thin lock. This thread has called Object.wait() on an object. The thread will remain there until some other thread sends a notification on that object.
This thread has called java.lang.Thread.sleep(). This thread has called java.util.concurrent.locks.LockSupport.park(). The threads execution has been suspended by java.lang.Thread.suspend() or a JVMTI/JVMPI agent call

suspended

Analyzing thread dumps

Classic deadlock
Look for the threads waiting for monitor entry: For eg: "ExecuteThread: '95' for queue: 'default'" daemon prio=5 tid=0x411cf8 nid=0x6c waiting for monitor entry [0xd0f80000..0xd0f819d8] at weblogic.common.internal.ResourceAllocator.release(ResourceAllocator.java:766) at weblogic.jdbc.common.internal.ConnectionEnv.destroy(ConnectionEnv.java:590) The above thread is waiting to acquire lock on ResourceAllocator object. The next step is to identify the thread that is holding the ResourceAllocator object
"ExecuteThread: '0' for queue: '__weblogic_admin_rmi_queue'" daemon prio=5 tid=0x41b978 nid=0x77 waiting for monitor entry [0xd0480000..0xd04819d8] at weblogic.jdbc.common.internal.ConnectionEnv.getPrepStmtCacheHits(ConnectionEnv.java:174) at weblogic.common.internal.ResourceAllocator.getPrepStmtCacheHitCount (ResourceAllocator.java:1525)

This thread is holding lock on ResourceAllocator object, but is waiting for ConnectionEnv object. This is a classic deadlock.

Analyzing thread dumps

Threads in wait()

A sample dump:
"ExecuteThread: '10' for queue: 'SERV_EJB_QUEUE'" daemon prio=5 tid=0x005607f0 nid=0x30 in Object.wait() [83300000..83301998] at java.lang.Object.wait(Native Method) - waiting on <0xc357bf18> (a weblogic.ejb20.pool.StatelessSessionPool) at weblogic.ejb20.pool.StatelessSessionPool.waitForBean(StatelessSessionPool.java:222) The above thread would come out of wait() under two conditions (depending on application logic) 1) One of the thread available in the execute queue pool would call notify() on this object when an instance is available. (If the wait() is indefinite). This can cause the thread to hang for ever if server never does a notify() to this object. 2) If the timeout exceeds, the thread would throw an exception and back to execute queue thread pool.

Samples

Questions?