Troubleshooting Performance Issues with Thread Dumps
Uncover deadlocks, thread leaks and decode your program behaviour with thread dumps.
Most developers improve the performance and monitor their Java Virtual Machines (JVM) by just watching the program logs, CPU, memory metrics, and, in some cases, response times. While these metrics are very useful, they do not provide deeper insights into what’s happening under the hood.
To address this, the jstack utility, introduced in JDK8, provides detailed stack traces of all threads in a JVM (including Java and VM threads). Thread dumps (generated by jstack) detail the program’s threads and can reveal bottleneck problems, deadlocks, or thread leaks.
What is a thread dump?
A thread dump is a snapshot of a JVM that details the threads’ states, information and stack traces.
"Quarkus Main Thread" #196 [62479] prio=5 os_prio=31 cpu=25.02ms elapsed=152.66s tid=0x00000001167c9000 nid=62479 waiting on condition [0x000000017229d000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.5/Native Method)
- parking to wait for <0x0000000781266890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at ...
As shown in the previous code snippet, the thread details are compiled into 6 main parts:
Thread Name: Quarkus Main Thread
Thread Type and Priority: prio=5 os_prio=5
Java Thread ID: tid=0x00000001167c9000 - This is the ID returned by Thread.getId()
Native Thread ID: nid=62479
Java Thread State - Six different thread states can be found in the report. For more details, please read the Javadocs for Thread.
New for threads that have not started.
Runnable for threads that are executing in the JVM.
Blocked for threads that are blocked for a monitor lock.
Waiting for threads that are waiting indefinitely for another thread to perform.
Parking: When a thread is waiting (parking) means that the thread is disabled for thread scheduling purposes unless the permit is available.
Sleeping: When a thread is waiting (sleeping) means that it cannot be notified by another thread (notify or notifyAll). The thread will be available once the sleep period is over.
Timed Waiting is similar to the waiting state, but in this case, the thread is waiting for another thread to perform in a specific waiting time.
Terminated for threads that have terminated their execution.
waiting on condition [0x000000017229d000] java.lang.Thread.State: WAITING (parking)Java Thread Stack Trace
at jdk.internal.misc.Unsafe.park(java.base@21.0.5/Native Method)
- parking to wait for <0x0000000781266890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at ...How do you collect Java thread dumps?
Thread dumps can be collected using jstack utility that was introduced in Java 8.
$ jstack <your-application-pid>Why are they important?
Thread dumps are crucial because they provide actionable insights for the engineering team to work on the program's performance and stability.
Diagnosing service degradation (slower response times or loss of service) can be achieved by analyzing the thread states. Usually, applications that have database connections may have some threads doing database queries and/or waiting to get their database connection.
Thread dumps automatically detect deadlocks and point out the affected classes.
Identify thread leaks (when too many threads are created to perform a task or when the threads are not terminated properly).
Small Demo
Clone the project, java-thread-dump-poc, review the README (details of the dependencies) and run the program using ./mvnw quarkus:dev
While the program is running, query the endpoints:
/hello for a basic interaction with the program thread,
/thread for starting a thread on the program.
/deadlock for creating a deadlock on the program
A pid is required to perform a thread dump using jstack. Find your program pid by running ps aux | grep java . And then, jstack <your-pid>
The README file contains the thread dumps for each request. The thread dump for the deadlock request contains details about the deadlock itself.
"Thread-55" #116 [51463] daemon prio=5 os_prio=31 cpu=0.48ms elapsed=18.30s tid=0x00000001470a0a00 nid=51463 waiting on condition [0x00000001733e6000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.5/Native Method)
- parking to wait for <0x00000007899c9d10>
"Thread-56" #117 [65287] daemon prio=5 os_prio=31 cpu=0.35ms elapsed=18.30s tid=0x0000000117cc5c00 nid=65287 waiting on condition [0x00000001735f2000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.5/Native Method)
- parking to wait for <0x00000007899c9ce0>
Java stack information for the threads listed above:
===================================================
"Thread-55":
at jdk.internal.misc.Unsafe.park(java.base@21.0.5/Native Method)
- parking to wait for <0x00000007899c9d10> (a java.util.concurrent.locks.ReentrantLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.5/LockSupport.java:221)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.5/AbstractQueuedSynchronizer.java:754)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.5/AbstractQueuedSynchronizer.java:990)
at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@21.0.5/ReentrantLock.java:153)
at java.util.concurrent.locks.ReentrantLock.lock(java.base@21.0.5/ReentrantLock.java:322)
at org.andrelizardo.DeadlockThread.run(DeadlockThread.java:19)
"Thread-56":
at jdk.internal.misc.Unsafe.park(java.base@21.0.5/Native Method)
- parking to wait for <0x00000007899c9ce0> (a java.util.concurrent.locks.ReentrantLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.5/LockSupport.java:221)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.5/AbstractQueuedSynchronizer.java:754)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.5/AbstractQueuedSynchronizer.java:990)
at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@21.0.5/ReentrantLock.java:153)
at java.util.concurrent.locks.ReentrantLock.lock(java.base@21.0.5/ReentrantLock.java:322)
at org.andrelizardo.DeadlockThread.run(DeadlockThread.java:19)
Found 1 deadlock.