<commit_before><html>
<title>TDA - Thread Dump Analyzer Tutorial</title>
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="Style">
<body>
<h2>TDA - Thread Dump Analyzer Tutorial</h2>
<h3>Introduction</h3>
<p>This is a short tutorial on how to use the TDA Thread Dump Analyzer. TDA parses your log files and displays all found thread dumps and class histograms reported from a Sun JVM &gt;= 1.4.x. Class Histograms are not included in
 the thread dumps by default but need a special JVM-Flag to be dumped with the thread dump. As TDA does everything offline and thread dumps have very low impact on the VM (including the class histogram option), it can be used for production environments. Note: There is a bug in 1.5 preventing the class histogram to be printed if any other then the concurrent mark sweep gc is used.
</p><p>
For having the class histograms included you need to provide the -XX:+PrintClassHistogram option.
</p>
<h3>Sample Parsing Session</h3>
<p>Open the log file you want to analyse, TDA will search for all thread dumps in this log file and displays them in a tree. As the Sun JVM thread dump lacks any date information when the dump was requested, you can provide a regular expression in the <b>Preferences</b> which is then used for parsing the lines before the thread dump to get a time stamp (see below).</p>
<p>For each thread dump TDA sums up all found threads and all monitors found in the threads, it also groups threads waiting on, for and locking threads.</p> 
<p><img src="threaddumps.png" border=0></p>
<p>If you want to focus on a lock found in a thread you can just mark it, expand the <b>Monitor</b> node of the thread dump and right-click on it and select the tree-search. Paste the marked monitor into the search dialog to go to the information about this monitor. You will get all waiting and all locking threads. Starting here you can easily see if a thread is hanging and holding lock which a lot of other threads are waiting for.
<p><img src="monitor.png" border=0></p>
<p>If you added the -XX:+PrintClassHistogram to the VM-Parameters you will also see the class histogram for a thread dump, presented as node of the dump. Here you can examine all objects in the heap at the time of the thread dump. You can sort this view and filter it using the <b>Filter Expression</b>.</p>
<p><img src="histogram.png" border=0></p>
<p>If you use the loggc option to log the garbage collection information into a different log file, the heap dump will go into this log file instead of standard out and the TDA will not find it. To add the heap dump you need to use <b>Open loggc file...</b> in the <b>File</b> menu. TDA will then parse this file backward and adds the found heap dumps to the thread dumps starting with the last dump. You can added multiple loggc-files, TDA will the continue after the last one where it added a heap dump with the last loggc-logfile. If you right click on a thread dump you also can start adding the heap dumps from this thread dump backward.
</p>
<p>If your log file contains a lot of thread dumps done during one session of the VM currently analyzed, you can use <b>Find long running threads</b> in the tools menu to extract long running threads from the thread dumps <i>to be done</i>.</p>

<h3>Setting regular expression for timestamp parsing</h3>
<p>Because the Sun JDK doesn't provide any time information when dumping threads a regular expression time stamp parsing for each line in the log file is included. TDA matches every line to the regular expression given in the preferences and stores the first matching group if the regular expression matches the line. It then takes the last match found as time stamp for the next thread dump to get a time stamp nearest to the thread dump. An example for such a regular expression is <i>(\d\d\/\d\d\/\d\d\s\d\d:\d\d:\d\d).*</i>. This expression matches lines like <i>06/02/14 14:54:04       at java.lang.Thread.run(Thread.java:534)</i>. TDA then takes the first capturing group as time stamp. In this example the \d is for digits and the capturing group is included in the brackets and matches <i>06/02/14 14:54:04</i>. This is then stored as timestamp for the next thread dump. </p>
<p>If the timestamp is stored in milliseconds since 1970 there is a checkbox in the preferences <i>Parsed timestamp is a long representing msecs since 1970</i> to tell TDA to convert the parsed time stamp from milliseconds into a human readable time stamp.</p> See References for detailed information about the regular expressions in java.

<h3>References</h3>
<p>For further offline analysis you can use the GC-Viewer found below. If you want to go one step further and do an online analysis with low impact on your (production) environment, you should have a look at the Glassbox Inspector.</p> 
<ul>
<li><a href="http://www.tagtraum.com/gcviewer.html">GC-Viewer</a>
<li><a href="http://glassbox-inspector.dev.java.net/">Glassbox Inspector</a>
<li><a href="http://java.sun.com/j2se/1.4.2/docs/api/index.html">Java API Specification 1.4.2</a> (for regular expressions look for <b>Pattern</b> class.)
</ul>
</body>
</html><commit_msg>added info about fetching a thread dump<commit_after><html>
<title>TDA - Thread Dump Analyzer Tutorial</title>
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="Style">
<body>
<h2>TDA - Thread Dump Analyzer Tutorial</h2>
<h3>Introduction</h3>
<p>This is a short tutorial on how to use the TDA Thread Dump Analyzer. TDA parses your log files and displays all found thread dumps and class histograms reported from a Sun JVM &gt;= 1.4.x. Class Histograms are not included in
 the thread dumps by default but need a special JVM-Flag to be dumped with the thread dump. As TDA does everything offline and thread dumps have very low impact on the VM (including the class histogram option), it can be used for production environments. Note: There is a bug in 1.5 preventing the class histogram to be printed if any other then the concurrent mark sweep gc is used.
</p><p>
For having the class histograms included you need to provide the -XX:+PrintClassHistogram option.
</p>
<h3>Obtaining a thread dump</h3>
For getting a thread dump, you need to send a kill-Signal to the Java VM, for this you need the PID of the VM and 
the do a <pre>kill -QUIT &lt;PID&gt;</pre> on a Unix Environment. If you use Windows you need a special tool for accomplishing 
this, SendSignal, see References for this. It does the same like <pre>kill -QUIT</pre> on Windows-Systems. Beware: if you're using the remote connection on windows, you need to start the connection it with <pre>mstsc.exe /console</pre> otherwise you wont have sufficient rights to use the tool. The JDK will then write the thread-dump to std-out. E.g. if your application is running in a tomcat, this would be catalina.out in a default tomcat installation.
<h3>Sample Parsing Session</h3>
<p>Open the log file you want to analyse, TDA will search for all thread dumps in this log file and displays them in a tree. As the Sun JVM thread dump lacks any date information when the dump was requested, you can provide a regular expression in the <b>Preferences</b> which is then used for parsing the lines before the thread dump to get a time stamp (see below).</p>
<p>For each thread dump TDA sums up all found threads and all monitors found in the threads, it also groups threads waiting on, for and locking threads.</p> 
<p><img src="threaddumps.png" border=0></p>
<p>If you want to focus on a lock found in a thread you can just mark it, expand the <b>Monitor</b> node of the thread dump and right-click on it and select the tree-search. Paste the marked monitor into the search dialog to go to the information about this monitor. You will get all waiting and all locking threads. Starting here you can easily see if a thread is hanging and holding lock which a lot of other threads are waiting for.
<p><img src="monitor.png" border=0></p>
<p>If you added the -XX:+PrintClassHistogram to the VM-Parameters you will also see the class histogram for a thread dump, presented as node of the dump. Here you can examine all objects in the heap at the time of the thread dump. You can sort this view and filter it using the <b>Filter Expression</b>.</p>
<p><img src="histogram.png" border=0></p>
<p>If you use the loggc option to log the garbage collection information into a different log file, the heap dump will go into this log file instead of standard out and the TDA will not find it. To add the heap dump you need to use <b>Open loggc file...</b> in the <b>File</b> menu. TDA will then parse this file backward and adds the found heap dumps to the thread dumps starting with the last dump. You can added multiple loggc-files, TDA will the continue after the last one where it added a heap dump with the last loggc-logfile. If you right click on a thread dump you also can start adding the heap dumps from this thread dump backward.
</p>
<p>If your log file contains a lot of thread dumps done during one session of the VM currently analyzed, you can use <b>Find long running threads</b> in the tools menu to extract long running threads from the thread dumps <i>to be done</i>.</p>

<h3>Setting regular expression for timestamp parsing</h3>
<p>Because the Sun JDK doesn't provide any time information when dumping threads a regular expression time stamp parsing for each line in the log file is included. TDA matches every line to the regular expression given in the preferences and stores the first matching group if the regular expression matches the line. It then takes the last match found as time stamp for the next thread dump to get a time stamp nearest to the thread dump. An example for such a regular expression is <i>(\d\d\/\d\d\/\d\d\s\d\d:\d\d:\d\d).*</i>. This expression matches lines like <i>06/02/14 14:54:04       at java.lang.Thread.run(Thread.java:534)</i>. TDA then takes the first capturing group as time stamp. In this example the \d is for digits and the capturing group is included in the brackets and matches <i>06/02/14 14:54:04</i>. This is then stored as timestamp for the next thread dump. </p>
<p>If the timestamp is stored in milliseconds since 1970 there is a checkbox in the preferences <i>Parsed timestamp is a long representing msecs since 1970</i> to tell TDA to convert the parsed time stamp from milliseconds into a human readable time stamp.</p> See References for detailed information about the regular expressions in java.

<h3>References</h3>
<p>For further offline analysis you can use the GC-Viewer found below. If you want to go one step further and do an online analysis with low impact on your (production) environment, you should have a look at the Glassbox Inspector.</p> 
<ul>
<li><a href="http://www.tagtraum.com/gcviewer.html">GC-Viewer</a>
<li><a href="http://glassbox-inspector.dev.java.net/">Glassbox Inspector</a>
<li><a href="http://java.sun.com/j2se/1.4.2/docs/api/index.html">Java API Specification 1.4.2</a> (for regular expressions look for <b>Pattern</b> class.)
<li><a href="http://www.latenighthacking.com/projects/2003/sendSignal/">SendSignal</a>
</ul>
</body>
</html>
{'fraction_non_alphanumeric': 0.07645369705671214, 'fraction_numerical': 0.011127063890882987, 'mean_word_length': 4.322349570200573, 'pattern_counts': {'":': 0, '<': 178, '<?xml version=': 0, '>': 178, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2092685', 'n_tokens_mistral': 3147, 'n_tokens_neox': 3001, 'n_words': 1738}