[SOLVED] Obtaining Thread Dumps from the OTM Java JVM Processes

[chrisplough]

Here's some quick instructions on obtaining thread dumps from the OTM Java JVM processes, including WebLogic and Tomcat.

These instructions differ slightly from those provided by Oracle support and tend to work better. In OTM / GC3 v4.x, I integrated the G-Log software with the JSW (Java Service Wrapper) software, which provides a cleaner way to kick off thread dumps and is less likely to kill the OTM server processes.

OTM Thread Dumps: In order to get thread dumps from OTM for Oracle Support, do the following.

Notes:

• Any time you run a thread dump, there is a minor chance that the OTM instance will die. Be prepared for this possibility, especially if running these against a production instance.
• Run the thread dumps simultaneously on the OTM Web and App servers, so that they can be compared directly against one another.
• Run 3 thread dumps on each server, separated by 30 seconds each. This will provide greater detail to support, as they can then tell which processes/threads are long-running and which are simply being processed through at that instant.

OTM App Server

1. Login to the server and change directories to /weblogic/config/gc3domain
2. Run

   ./weblogic.sh dump

3. Wait 30 seconds and repeat step 2.
4. Repeat until you’ve done a total of 3 thread dumps.
5. Copy the /logs/weblogic/console.log file and send to support – this log file contains the thread dumps.

OTM Web Server

1. Login to the server and change directories to /tomcat/bin
2. Run

   ./tomcat.sh dump

3. Wait 30 seconds and repeat step 2.
4. Repeat until you’ve done a total of 3 thread dumps.
5. Copy the /logs/tomcat/console.log file and send to support – this log file contains the thread dumps.

I hope this helps!

--Chris