<commit_before>:imagesdirs: .

== Examples

These examples all all based on some https://github.com/binaryfoo/lagotto/blob/master/log_samples/q2.log[sample data] kindly provided by https://twitter.com/apr[@apr].

If you want to try them out:

. git clone https://github.com/binaryfoo/lagotto.git
. cd lagotto
. mvn package -DskipTests
. ./lago <... example ...>

=== Filtering

Only output entries with an MTI (field 0) of 2100 using --field.

    LAGO: log_samples/q2.log --field mti=2100 -n 1

The --field option take an argument of the form path=value. Path can be dot delimited like 48.1.1. The -n limits the number of entries output. Like in 'nix head and tail.

To make the remaining example output shorter let's introduce the idea tabular output (--table). List out just the fields you're interested in.

    LAGO: log_samples/q2.log --table time,2,11 -f mti=2100 -n 1

Whilst comma and tab separated data might be good for import into Excel/Google Docs/Libre Office/MySQL an ASCII table like mysql's command line client is a bit easier for humans to parse on the command line.

    LAGO: log_samples/q2.log -o ascii --table time,2,11 -f mti=2100 -n 1

Back to filtering, you can filter using somewhat familiar operators: <, >, !=, ~, and ~/regex/. The last two are contains and http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html[regex].

To show log events that took more than 300ms to receive or send use >. The following two receives aren't really interesting. A send with a non zero lifespan usually means a GC pause or TCP back pressure.

    LAGO: log_samples/q2.log -o ascii --table time,msgType,lifespan,src -f 'lifespan>300'

You can also use --grep to just search the log entries as text.

    LAGO: log_samples/q2.log --grep suspicious -o ascii -t time,63,src

=== Aggregation

Some rather basic SQL like group by and aggregation operators are supported.

Show a table of message type and count.

    LAGO: log_samples/q2.log -o ascii -t mti,count

Count rows matching a condition (like approved).

    LAGO: log_samples/q2.log -o ascii -t 'mti,count,count(39=0000)'

Combine with filtering to get the spread of response codes for auths.

    LAGO: log_samples/q2.log -o ascii -t '39,count' --field '39!=0000' --field mti=2110

Bucketing results by time. You can use a joda time http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html[format] or a few extras to aggregate by 10 minute (HH:m0) or 10 second (HH:mm:s0) buckets.

    LAGO: log_samples/q2.log -o ascii -t 'time(HH:mm:s0),count'

The results aren't that interesting with such a small data set.

TODO group_concat

TODO group_sample

=== Pairing Requests & Responses

Match responses to requests using --pair. Show one line per pair including the round trip time (rtt): response - request time

    LAGO: log_samples/q2.log -o ascii -t time,mti,11,rtt --pair

=== Cleaning Data

Tidy up a field a little using a regex replacement. This feature is probably somewhat gratuitous.

    LAGO: log_samples/q2.log -o ascii -t 'time(HH:mm),mti,11(/^0+//),rtt' --pair

=== Sorting

Sort using a field with --sort rtt

    LAGO: log_samples/q2.log -o ascii -t 'time,mti,11(/^0+//),rtt' --pair --sort rtt

Or sort descending using --sort rtt desc

    LAGO: log_samples/q2.log -o ascii -t 'time,mti,11(/^0+//),rtt' --pair --sort 'rtt desc'

=== More Filtering

Show only pairs where the response took more than 100ms using --field

    LAGO: log_samples/q2.log -o ascii -t time,mti,11,rtt --pair --field rtt>100

Remember you'll need to use quotes 'rtt>100' or a slash rtt\>100 to escape the '>' when using bash.

Add the file name and line number to get more detail (or check lago is not lying)

    LAGO: log_samples/q2.log -o ascii -t time,mti,11,rtt,src --pair --field rtt>100

Use some aggregate operators min(), max() and avg() to get basic stats about the round trip times

    LAGO: log_samples/q2.log -o ascii -t mti,min(rtt),max(rtt),avg(rtt),count --pair

Find delays of more than 100ms in a message sequence

    LAGO: log_samples/q2.log -o ascii -t time,delay,src --field delay>100

Show a table of exceptions. Not very interesting with only one exception but useful when there are many

    LAGO: log_samples/q2.log -o ascii -t exception,src --field exception!=

Show a breakdown of message types

    LAGO: log_samples/q2.log -o ascii -t msgType,count

=== Dictionaries

TODO Convert a https://github.com/jpos/jPOS-CMF/blob/master/src/docx/result_codes.xml[snippet] of jPOS' CMF into a dictionary to show how translation works

    LAGO: log_samples/q2.log -o ascii -t 'translate(39),39,count' --field mti=2110

=== Histograms

You can output an http://hdrhistogram.org/[HDR Histogram] using --histogram. Has some use for http://hdrhistogram.github.io/HdrHistogram/plotFiles.html[plotting] response times (rtt).

    LAGO: log_samples/q2.log --pair --histogram rtt

=== JSON export

Export each log entry as a single line of JSON. The driver idea here was to allow import into Apache Spark for query using https://spark.apache.org/sql/[Spark SQL].

    LAGO: log_samples/q2.log --field mti=2100 -n 1 --json

=== Digest format

Output a slightly less wordy output. Supports translation of field paths to names using a dictionary (TODO explain...).

    LAGO: log_samples/q2.log --field mti=2100 -n 1 --digest

=== Gnuplot script generation

    LAGO: log_samples/q2.log --pair --table time,rtt --gnuplot rtts

Produces a .csv file to be plotted
----
include::rtts.csv[]
----
A gnuplot script to plot the .csv data
----
include::rtts.gp[]
----
Executes the gnuplot script to produce a chart

image::rtts.svg[Example plot]

=== When you need more

See https://github.com/binaryfoo/lagotto/tree/master/src/main/scala/io/github/binaryfoo/lagotto/examples[examples] or the sbt shell script https://github.com/binaryfoo/lagotto-examples/tree/master/src/main/script[examples].

=== log4j

TODO: Might be a can of worms given the variety of possible input formats...

[small]#The output for all examples is generated by https://github.com/binaryfoo/lagotto/blob/master/src/test/scala/io/github/binaryfoo/lagotto/RunExamples.scala[RunExamples.scala].#<commit_msg>Add garbage collection examples<commit_after>:imagesdirs: .

== Examples

These examples all all based on some https://github.com/binaryfoo/lagotto/blob/master/log_samples/q2.log[sample data] kindly provided by https://twitter.com/apr[@apr].

If you want to try them out:

. git clone https://github.com/binaryfoo/lagotto.git
. cd lagotto
. mvn package -DskipTests
. ./lago <... example ...>

=== Filtering

Only output entries with an MTI (field 0) of 2100 using --field.

    LAGO: log_samples/q2.log --field mti=2100 -n 1

The --field option take an argument of the form path=value. Path can be dot delimited like 48.1.1. The -n limits the number of entries output. Like in 'nix head and tail.

To make the remaining example output shorter let's introduce the idea tabular output (--table). List out just the fields you're interested in.

    LAGO: log_samples/q2.log --table time,2,11 -f mti=2100 -n 1

Whilst comma and tab separated data might be good for import into Excel/Google Docs/Libre Office/MySQL an ASCII table like mysql's command line client is a bit easier for humans to parse on the command line.

    LAGO: log_samples/q2.log -o ascii --table time,2,11 -f mti=2100 -n 1

Back to filtering, you can filter using somewhat familiar operators: <, >, !=, ~, and ~/regex/. The last two are contains and http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html[regex].

To show log events that took more than 300ms to receive or send use >. The following two receives aren't really interesting. A send with a non zero lifespan usually means a GC pause or TCP back pressure.

    LAGO: log_samples/q2.log -o ascii --table time,msgType,lifespan,src -f 'lifespan>300'

You can also use --grep to just search the log entries as text.

    LAGO: log_samples/q2.log --grep suspicious -o ascii -t time,63,src

=== Aggregation

Some rather basic SQL like group by and aggregation operators are supported.

Show a table of message type and count.

    LAGO: log_samples/q2.log -o ascii -t mti,count

Count rows matching a condition (like approved).

    LAGO: log_samples/q2.log -o ascii -t 'mti,count,count(39=0000)'

Combine with filtering to get the spread of response codes for auths.

    LAGO: log_samples/q2.log -o ascii -t '39,count' --field '39!=0000' --field mti=2110

Bucketing results by time. You can use a joda time http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html[format] or a few extras to aggregate by 10 minute (HH:m0) or 10 second (HH:mm:s0) buckets.

    LAGO: log_samples/q2.log -o ascii -t 'time(HH:mm:s0),count'

The results aren't that interesting with such a small data set.

TODO group_concat

TODO group_sample

=== Pairing Requests & Responses

Match responses to requests using --pair. Show one line per pair including the round trip time (rtt): response - request time

    LAGO: log_samples/q2.log -o ascii -t time,mti,11,rtt --pair

=== Cleaning Data

Tidy up a field a little using a regex replacement. This feature is probably somewhat gratuitous.

    LAGO: log_samples/q2.log -o ascii -t 'time(HH:mm),mti,11(/^0+//),rtt' --pair

=== Sorting

Sort using a field with --sort rtt

    LAGO: log_samples/q2.log -o ascii -t 'time,mti,11(/^0+//),rtt' --pair --sort rtt

Or sort descending using --sort rtt desc

    LAGO: log_samples/q2.log -o ascii -t 'time,mti,11(/^0+//),rtt' --pair --sort 'rtt desc'

=== More Filtering

Show only pairs where the response took more than 100ms using --field

    LAGO: log_samples/q2.log -o ascii -t time,mti,11,rtt --pair --field rtt>100

Remember you'll need to use quotes 'rtt>100' or a slash rtt\>100 to escape the '>' when using bash.

Add the file name and line number to get more detail (or check lago is not lying)

    LAGO: log_samples/q2.log -o ascii -t time,mti,11,rtt,src --pair --field rtt>100

Use some aggregate operators min(), max() and avg() to get basic stats about the round trip times

    LAGO: log_samples/q2.log -o ascii -t mti,min(rtt),max(rtt),avg(rtt),count --pair

Find delays of more than 100ms in a message sequence

    LAGO: log_samples/q2.log -o ascii -t time,delay,src --field delay>100

Show a table of exceptions. Not very interesting with only one exception but useful when there are many

    LAGO: log_samples/q2.log -o ascii -t exception,src --field exception!=

Show a breakdown of message types

    LAGO: log_samples/q2.log -o ascii -t msgType,count

=== Dictionaries

TODO Convert a https://github.com/jpos/jPOS-CMF/blob/master/src/docx/result_codes.xml[snippet] of jPOS' CMF into a dictionary to show how translation works

    LAGO: log_samples/q2.log -o ascii -t 'translate(39),39,count' --field mti=2110

=== Histograms

You can output an http://hdrhistogram.org/[HDR Histogram] using --histogram. Has some use for http://hdrhistogram.github.io/HdrHistogram/plotFiles.html[plotting] response times (rtt).

    LAGO: log_samples/q2.log --pair --histogram rtt

=== JSON export

Export each log entry as a single line of JSON. The driver idea here was to allow import into Apache Spark for query using https://spark.apache.org/sql/[Spark SQL].

    LAGO: log_samples/q2.log --field mti=2100 -n 1 --json

=== Digest format

Output a slightly less wordy output. Supports translation of field paths to names using a dictionary (TODO explain...).

    LAGO: log_samples/q2.log --field mti=2100 -n 1 --digest

=== Gnuplot script generation

    LAGO: log_samples/q2.log --pair --table time,rtt --gnuplot rtts

Produces a .csv file to be plotted
----
include::rtts.csv[]
----
A gnuplot script to plot the .csv data
----
include::rtts.gp[]
----
Executes the gnuplot script to produce a chart

image::rtts.svg[Example plot]

=== When you need more

See https://github.com/binaryfoo/lagotto/tree/master/src/main/scala/io/github/binaryfoo/lagotto/examples[examples] or the sbt shell script https://github.com/binaryfoo/lagotto-examples/tree/master/src/main/script[examples].

=== log4j

TODO: Might be a can of worms given the variety of possible input formats...

=== Garbage Collection logs

    lago --in-format gc-v2 --table 'datetime,pause,heapBefore,heapAfter,heapMax,PSYoungGenBefore,PSYoungGenAfter,PSYoungGenMax,ParOldGenBefore,ParOldGenAfter,ParOldGenCapacityBefore' gc.log --sort datetime --gnuplot gc --plot '(2)(3,4,5)(6,7,8)(9,10,11)' --ui -f 'type!=GC--'

    lago --in-format gc-v2 --table 'datetime,pause,heapBefore,heapAfter,heapMax,PSYoungGenBefore,PSYoungGenAfter,PSYoungGenMax,ParOldGenBefore,ParOldGenAfter,ParOldGenCapacityBefore,type,ordinal(type)' gc.log --sort datetime --gnuplot gc --plot '(2)(3,4,5)(6,7,8)(9,10,11)'

[small]#The output for all examples is generated by https://github.com/binaryfoo/lagotto/blob/master/src/test/scala/io/github/binaryfoo/lagotto/RunExamples.scala[RunExamples.scala].#
{'fraction_non_alphanumeric': 0.11141115654656276, 'fraction_numerical': 0.024226525731039273, 'mean_word_length': 3.664267722202231, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 23, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2189382', 'n_tokens_mistral': 4594, 'n_tokens_neox': 4229, 'n_words': 1566}