Diagnosing a JVM Crash!

JVM Crash

JVM crash is one of the toughest problem professional Java developers face. In case of a JVM crash, the operating system creates a core dump file which is a memory snapshot of a running process. A core dump is created by the operating system when a fatal or unhandled error like signal or system exception occurs. For professional Java developers, it is more common to report such errors to a JVM vendor because for a Java developer it is not common to troubleshoot native OS processes. That adds a significant wait period for troubleshooting and finding a fix for the crash. OpenJDK provides necessary tools for Java developers to diagnose such problems. With these tools we don’t need to wait for the JDK vendor, we can troubleshoot and find the root cause of the crash ourselves.

 

Common causes of JVM Crash

JVM crash can occur due to a bug

1. in the Java HotSpot VM
2. in a system library
3. in a Java SE library or an API
4. in application native code
5. in the operating system (OS)

The majority of the above bugs falls under 2 categories:

1. Running out of native memory. This is caused when native memory is exhausted, read more about native memory allocation here
2. Segmentation faults. These are caused by a program trying to read or write an illegal memory location.

 

Creating core dump

On JVM crash a core dump is created by the Operating system. In some cases when JVM is hanged we may want to create a memory snapshot of running JVM to analyze this in detail, for this we can create a core dump of the JVM process manually. Let’s take a look at the command we need to run to create a core dump, and we will try to analyze that in the next section.

 

Java program to analyze in this post

This is a sample bad Java program Test.java that we will be using to create a core dump. For all our examples we will be using Java 18, as of writing this post it is built using JDK master branch. This post can help you to build JDK from the source.

public class Test
{
public static void main(String[] args)
{
Object object;

while(true)
{
object = new Object();
}
}
}

 

Operating system changes to create a core dump

Following are OS-level changes required on my Linux machine to enable creating a core dump file.

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
ulimit -c unlimited

In case you find difficulty generating core dump, refer detailed documentation from here.

 

Creating core dump

Commands to run a Java program for which we will create a core dump file.

/home/vipin/githubprojects/jdk/build/linux-x86_64-server-release/images/jdk/bin/javac Test.java
/home/vipin/githubprojects/jdk/build/linux-x86_64-server-release/images/jdk/bin/java Test

Here is command to fetch PID of running Java process, to read more about jcmd see this post

$ jcmd
78480 Test
78501 jdk.jcmd/sun.tools.jcmd.JCmd

Following is a list of commands required to create a core dump file using GNU debugger gdb

gdb
attach 78480
gcore jvm_dump.core
detach
quit

 

Analyzing core dump

A core dump file created by OS is more friendly for C and C++ developers, they use tools like gdb to get information out of the core dump files. OpenJDK provides CLI tool jhsdb which can be used to troubleshoot JVM core dump. Tools like gdb have no knowledge about Java data structures, while jhsdb is a Java developer’s friendly tool to analyze core dump files. jhsdb has a lot of features available, some important ones are

1. we can use jmap mode for memory information
2. jstack mode for the stack trace
3. jinfo mode for command-line flags and the system properties.

For more details on all available modes, you can read about jhsdb here.

 

Let’s try to connect jhsdb with the core dump file jvm_dump.core created in the previous section. In practice, it could be either a core dump file generated from a crash or manually by the command line.

We will try to analyze the core dump using jhsdb in 2 modes

1. clhsdb (Command line debugger)
2. hsdb (GUI debugger)

 

jhsdb command line debugger clhsdb

Following command opens core dump in command line mode for further analysis

jhsdb clhsdb --core jvm_dump.core --exe /home/vipin/githubprojects/jdk/build/linux-x86_64-server-release/images/jdk/bin/java
Opening core file, please wait...
hsdb>

Now that we have the hsdb prompt we can run the threads command to get a list of threads along with thread ids.

hsdb> threads
78481 main
State: IN_JAVA
Stack in use by Java: 0x00007f040bb67a10 .. 0x00007f040bb67a30
Base of Stack: 0x00007f040bb69000
Last_Java_SP: null
Last_Java_FP: null
Last_Java_PC: null
Thread id: 78481
...
78488 Reference Handler
State: BLOCKED
Stack in use by Java: 0x00007f03c8bfe940 .. 0x00007f03c8bfea88
Base of Stack: 0x00007f03c8c00000
Last_Java_SP: 0x00007f03c8bfe940
Last_Java_FP: 0x00007f03c8bfe990
Last_Java_PC: 0x00007f03ed47b50b
Thread id: 78488
...
...
...

Now let’s try to get a stack trace for thread id 78481

hsdb> where 78481
Thread 78481 Address: 0x00007f0404023b50

Java Stack Trace for main
Thread state = IN_JAVA
- public static void main(java.lang.String[]) @0x00007f03c90003a0 @bci = 60, line = 19, pc = 0x00007f03f5015e4d (Compiled; information may be imprecise)

In the above output, we can see only one method call in the stack trace, because Test.java has only the main method. A thread dump is useful when we have a deadlock situation, or we want to find out what all threads are doing in the hanged java application.

In the following command, we are trying to print a histogram from the core dump, which shows an unusually high number of instances for Object. Here is an example of using jhisto

hsdb> jhisto
Iterating over heap. This may take a while...
Object Histogram:

num      #instances   #bytes Class description
--------------------------------------------------------------------------
1:    933    7386016    int[]
2:    160637 2570192    java.lang.Object
3:    7546   366360 byte[]
4:    1432   176464 java.lang.Class
5:    7251   174024 java.lang.String
6:    1134   97736  java.lang.Object[]
7:    7  33032  char[]
8:    1032   33024  java.util.concurrent.ConcurrentHashMap$Node
...
...

We can use help to get a list of commands available clhsdb

hsdb> help
Available commands:
assert true | false
attach pid | exec core | remote_server
buildreplayjars [ all | app | boot ]  | [ prefix ]
class name
classes
detach
...
...
...
thread { -a | id }
threads
tokenize ...
type [ type [ name super isOop isInteger isUnsigned size ] ]
universe
verbose true | false
versioncheck [ true | false ]
vmstructsdump
where { -a | id }

 

jhsdb gui debugger hsdb

The following command opens a GUI window that provides an interactive option to see details of the core dump. By default, it opens a list of threads as well.

jhsdb hsdb --core jvm_dump.core --exe /home/vipin/githubprojects/jdk/build/linux-x86_64-server-release/images/jdk/bin/java

In the tools menu there are various options available to see details of the core dump, below is the result when selected “Show VM Version”

When we select Object Histogram option from tools menu below is output:

 

Conclusion

JVM crash is not a common problem but when it happens we may need to raise it with the JVM vendor and wait for a significant duration for the RCA/resolution. jhsdb is a very useful command for Java developers we can analyze JVM crash ourselves and take corrective action as well.

If you want to get amazing Java jobs, I wrote an ebook 5 steps to Best Java Jobs. You can download this step-by-step guide for free!

Resources

1. Java tools reference
2. Detailed post on jhsdb
3. Reasons for not Getting a Core File
4. Build JDK from source