How to get and analyze thread dumps

Introduction

A Java thread dump is a snapshot what every thread in the JVM is doing at a particular point in time. Each thread in the JVM is listed with it's name and id, it's current state and the Java call stack showing what monitor it has locked or is waiting on. This is especially useful if your Java application sometimes seems to hang when running under load, as an analysis of the dump will show where the threads are stuck, either by deadlock or other thread contention.

How to get a thread dump

Unix:

On most flavors of Unix kill -3 <pid> will generate a thread dump. To get the java process pid use ps -ef | grep java. The dump is directed to stdout so redirecting to a file is neccessary. Since 1.6 the Sun/Oracle JDK and openJDK has the jstack utility to do the same thing, run it from the command line and redirect to a file.

Windows:

If the application was started from a command window you can select the window and use CTRL+Breakbut the output will likely overflow the window. Jstack is available if your using the Sun/Oracle JDK and openJDK. Otherwise use PsExec.exe to execute SendSignal.exe. Also, if Tomcat is started as a service it is necessary to use SendSignal. 

How many thread dumps do I need

Often a single thread dump will reveal the problem, this is especially true with deadlocks where two or more threads are waiting for locks obtained by each other. But in other cases, such as threads waiting on other processes, like IO from a database, you won't be able to detect it with a single thread dump. In this case multiple thread dumps taken over time will show the same thread waiting for the process to complete during that time span.

The scripts attached at the bottom of this article will generate multiple thread dumps over time. Running the script from the command line, the default is 10 thread dumps with a 60 second delay between each dump, this is configurable from the command line. If there is only one Java process running on the machine the script will detect it and get the pid, otherwise you will have to uncomment the pid variable in the script and enter the pid at the command line.

Sometimes the issue is intermitent, occurring at random times. Since you want to gather the series of thread dumps before the issue pops up you can create a cron job on Unix systems or use the Task Scheduler on windows to run the script at regular intervals to obtain a two hour window of thread dumps (the script deletes any dump file older than two hours so it will not fill up the file system). A sample cron job is included with the scripts.

Analyzing the thread dumps

You can read a thread dump with a text editor, and many people use only this. Thread Dump Analyzer is a useful tool to aid in the process. It is a Java Swing application for analyzing thread dumps. It provides statistics about the thread dump, displays individual threads and gives information about locked monitors and waiting threads, flagging monitors that have many threads waiting on it. https://java.net/projects/tda.

A detailed description of how to analyze thread dumps is beyond the scope of this article.

However, there are a host of good articles on the web that expain the process in great detail. Here are two sources that I like:

• http://javaeesupportpatterns.blogspot.com/p/thread-dump-analysis.html
• http://web.archive.org/web/20130214060723/http://www.cubrid.org/blog/dev-platform/how-to-analyze-java-thread-dumps/

tread_dumps_0.zip

thread_dumps.zip