<commit_before>
Log Processing
==============

To properly generate log plots, you must enable VDL/Karajan logging.

You should check the scripts that you intend to use to determine
what log lines they require and ensure that you are generating
those lines via log4j.properties

Make sure log4.properties contains:
--------------------------------------
log4j.logger.swift=DEBUG
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=DEBUG
--------------------------------------

Normalize event times in the log to the run start time
------------------------------------------------------

* Generate the log, assuming the log is titled +swift-run.log+

------------------------------------------
make -f libexec/log-processing/makefile.implicit swift-run.plot.norm
------------------------------------------

Make a basic load plot from Coasters Cpu log lines
--------------------------------------------------

. Normalize the log.
. Build up a load data file:
+
------------------------------------------
./cpu-job-load.pl < swift-run.norm > load.data
------------------------------------------
. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
lines.zsh load.cfg load.eps load.data
------------------------------------------

Make a basic job completion plot from Coasters Cpu log lines
------------------------------------------------------------

. Normalize the log.

. Build up a completed data file:
+
------------------------------------------
./cpu-job-completed.pl < swift-run.norm > completed.data
------------------------------------------

. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
lines.zsh completed.cfg completed.eps completed.data
------------------------------------------

Make a basic Block allocation plot from Coasters Block log lines
----------------------------------------------------------------

. Normalize the log.

. Build up a block allocation data file:
+
------------------------------------------
./block-level.pl < swift-run.norm > blocks.data
------------------------------------------

. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
lines.zsh blocks.{cfg,eps,data}
------------------------------------------

Make a job runtime distribution plot from Coasters Cpu log lines
----------------------------------------------------------------

. Normalize the log.

. Build up a job runtime file:
+
------------------------------------------
./extract-times.pl < swift-run.norm > times.data
------------------------------------------

. Put the job runtimes into 1-second buckets:
+
------------------------------------------
./ buckets.pl 1 times.data > buckets.data
------------------------------------------

. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
lines.zsh buckets.cfg buckets.eps buckets.data
------------------------------------------
<commit_msg>minor change to the readme to use the pearl script to normalize the log<commit_after>
Log Processing
==============

To properly generate log plots, you must enable VDL/Karajan logging.

You should check the scripts that you intend to use to determine
what log lines they require and ensure that you are generating
those lines via log4j.properties

Make sure log4.properties contains:
--------------------------------------
log4j.logger.swift=DEBUG
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=DEBUG
--------------------------------------

Normalize event times in the log to the run start time
------------------------------------------------------

* Generate the log, assuming the log is titled +swift-run.log+

------------------------------------------
./normalize-log.pl file.contains.start.time swift-run.log > swift-run.norm
------------------------------------------

Make a basic load plot from Coasters Cpu log lines
--------------------------------------------------

. Normalize the log.
. Build up a load data file:
+
------------------------------------------
./cpu-job-load.pl < swift-run.norm > load.data
------------------------------------------
. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
lines.zsh load.cfg load.eps load.data
------------------------------------------

Make a basic job completion plot from Coasters Cpu log lines
------------------------------------------------------------

. Normalize the log.

. Build up a completed data file:
+
------------------------------------------
./cpu-job-completed.pl < swift-run.norm > completed.data
------------------------------------------

. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
lines.zsh completed.cfg completed.eps completed.data
------------------------------------------

Make a basic Block allocation plot from Coasters Block log lines
----------------------------------------------------------------

. Normalize the log.

. Build up a block allocation data file:
+
------------------------------------------
./block-level.pl < swift-run.norm > blocks.data
------------------------------------------

. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
lines.zsh blocks.{cfg,eps,data}
------------------------------------------

Make a job runtime distribution plot from Coasters Cpu log lines
----------------------------------------------------------------

. Normalize the log.

. Build up a job runtime file:
+
------------------------------------------
./extract-times.pl < swift-run.norm > times.data
------------------------------------------

. Put the job runtimes into 1-second buckets:
+
------------------------------------------
./ buckets.pl 1 times.data > buckets.data
------------------------------------------

. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
lines.zsh buckets.cfg buckets.eps buckets.data
------------------------------------------

{'fraction_non_alphanumeric': 0.4344262295081967, 'fraction_numerical': 0.0022068095838587644, 'mean_word_length': 0.9511070110701108, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5195828', 'n_tokens_mistral': 1341, 'n_tokens_neox': 1312, 'n_words': 445}