<commit_before>.TH LOGDISSECT 1
.SH NAME
logdissect - A robust CLI syslog forensics tool.

.SH SYNOPSIS
\fBlogdissect\fR [\fB-h\fR] [\fB--dhost \fIDHOST\fR] [\fB--grep \fIPATTERN\fR] [\fB--last \fILAST\fR] [\fB--process \fIPROCESS\fR] [\fB--protocol \fIPROTOCOL\fR] [\fB--range \fIRANGE\fR] [\fB--rdhost \fIDHOST\fR] [\fB--rgrep \fIPATTERN\fR] [\fB--rprocess \fIPROCESS\fR] [\fB--rshost \fISHOST\fR] [\fB--rsource \fISOURCE\fR] [\fB--shost \fISHOST\fR] [\fB--source \fISOURCE\fR] [\fB--outlog \fIOUTLOG\fR] [\fB--label \fILABEL\fR] [\fB--outjson \fIOUTJSON\fR] [\fB--version\fR] [\fB--verbose\fR] [\fB-s\fR] [\fB--list-parsers\fR] [\fB-p \fIPARSER\fR] [\fB-z\fR] [\fB-t \fITZONE\fR] [\fBfile \fR[\fBfile ...\fR]]

.SH DESCRIPTION
\fBlogdissect\fP is a tool for gaining insight into syslog files. It can merge multiple log files together and sort by timestamp, and filter the results by time range and other criteria. Results are output to the terminal by default, and can also be output a to standard syslog file, or to a JSON array along with some metadata.

.SH OPTIONS

  usage: \fBlogdissect\fR [\fBoptions\fR] \fBfile\fR [\fBfile\fR] ...
  
  positional arguments:
    \fBfile\fR                 specify input files
  
  optional arguments:
    \fB-h\fR, \fB--help\fR           show this help message and exit
    \fB--version\fR            show program's version number and exit
    \fB--verbose\fR            set verbose terminal output
    \fB-s\fR                   silence terminal output
    \fB--list-parsers\fR       return a list of available parsers
    \fB-p\fR \fIPARSER\fR            select a parser (default: syslog)
    \fB-z\fR, \fB--unzip\fR          include files compressed with gzip
    \fB-t\fR \fITZONE\fR             specify timezone offset to UTC (e.g. '+0500')

  
  morph options:
    \fB--dhost\fR \fIDHOST\fR        match a destination host
    \fB--grep\fR \fIPATTERN\fR       match a pattern
    \fB--last\fR \fILAST\fR          match a preceeding time period (5m/3h/2d/etc)
    \fB--process\fR \fIPROCESS\fR    match a source process
    \fB--protocol\fR \fIPROTOCOL\fR  match a protocol
    \fB--range\fR \fIRANGE\fR        match a time range <YYYYMMDDhhmm-YYYYMMDDhhmm>
    \fB--utc\fR                use UTC for range matching
    \fB--rdhost\fR \fIDHOST\fR       filter out a destination host
    \fB--rgrep\fR \fIPATTERN\fR      filter out a pattern
    \fB--rprocess\fR \fIPROCESS\fR   filter out a source process
    \fB--rprotocol\fR \fIPROTOCOL\fR match a protocol
    \fB--rshost\fR \fISHOST\fR       filter out a source host
    \fB--rsource\fR \fISOURCE\fR     filter out a log source
    \fB--shost\fR \fISHOST\fR        match a source host
    \fB--source\fR \fISOURCE\fR      match a log source

  
  output options:
    \fB--outlog \fIOUTLOG\fR      set the output file for standard log output
    \fB--label \fILABEL\fR        set label type for OUTLOG (fname|fpath)
    \fB--outjson \fIOUTJSON\fR    set the output file for JSON output


  ==== Available parsing modules: ====
  
  ciscoios        : cisco ios parsing module
  ldjson          : logdissect JSON parsing module
  syslog          : syslog (standard timestamp) parsing module
  syslogiso       : syslog (ISO timestamp) parsing module
  syslognohost    : syslog (standard timestamp, no host) parsing module
  tcpdump         : tcpdump terminal output parsing module
  windowsrsyslog  : windows rsyslog agent log parsing module

.SH EXAMPLES
logdissect --last 10m auth.log
.br
logdissect --last 1h --process systemd --process CRON messages
.br
logdissect -v --range 20160202020202-20170227213200 --label fpath messages debug apache2/error.log
.br
logdissect -s --outlog myaccess.log --grep 192.168.1.56 --last 30d --label fname /var/log/apache2/access.log /var/log/apache2/other_vhosts_access.log

.SH NOTES
1. metadata: logdissect uses file modification times to assign years to syslog date stamps. This allows it to parse logs that span more than one year without a problem. If you are copying log files, always use 'cp -p' (or 'cp --preserve=timestamps') and 'scp -p' to preserve original mtimes and other file metadata.

2. Re-parsing: If you are planning on parsing an output file back into logdissect at some point, using JSON is highly recommended. The JSON output module uses date stamps that include a year. Re-parsing a standard log output file will cause problems if the file has a different mtime than the original logs.

3. --range shortcuts: The range module will fill in your ranges with zeroes if they are shorter than 14 characters. If you want to get a range of 20170204120000 to 20170204130000, you can save time and use 2017020412 and 2017020413.<commit_msg>Update docs<commit_after>.TH LOGDISSECT 1
.SH NAME
logdissect - A robust CLI syslog forensics tool.

.SH SYNOPSIS
\fBlogdissect\fR [\fB-h\fR] [\fB--dhost \fIDHOST\fR] [\fB--grep \fIPATTERN\fR] [\fB--last \fILAST\fR] [\fB--process \fIPROCESS\fR] [\fB--protocol \fIPROTOCOL\fR] [\fB--range \fIRANGE\fR] [\fB--rdhost \fIDHOST\fR] [\fB--rgrep \fIPATTERN\fR] [\fB--rprocess \fIPROCESS\fR] [\fB--rshost \fISHOST\fR] [\fB--rsource \fISOURCE\fR] [\fB--shost \fISHOST\fR] [\fB--source \fISOURCE\fR] [\fB--outlog \fIOUTLOG\fR] [\fB--label \fILABEL\fR] [\fB--outjson \fIOUTJSON\fR] [\fB--version\fR] [\fB--verbose\fR] [\fB-s\fR] [\fB--list-parsers\fR] [\fB-p \fIPARSER\fR] [\fB-z\fR] [\fB-t \fITZONE\fR] [\fBfile \fR[\fBfile ...\fR]]

.SH DESCRIPTION
\fBlogdissect\fP is a tool for gaining insight into syslog files. It can merge multiple log files together and sort by timestamp, and filter the results by time range and other criteria. Results are output to the terminal by default, and can also be output a to standard syslog file, or to a JSON array along with some metadata.

.SH OPTIONS

  usage: \fBlogdissect\fR [\fBoptions\fR] \fBfile\fR [\fBfile\fR] ...
  
  positional arguments:
    \fBfile\fR                 specify input files
  
  optional arguments:
    \fB-h\fR, \fB--help\fR           show this help message and exit
    \fB--version\fR            show program's version number and exit
    \fB--verbose\fR            set verbose terminal output
    \fB-s\fR                   silence terminal output
    \fB--list-parsers\fR       return a list of available parsers
    \fB-p\fR \fIPARSER\fR            select a parser (default: syslog)
    \fB-z\fR, \fB--unzip\fR          include files compressed with gzip
    \fB-t\fR \fITZONE\fR             specify timezone offset to UTC (e.g. '+0500')

  
  morph options:
    \fB--dhost\fR \fIDHOST\fR        match a destination host
    \fB--grep\fR \fIPATTERN\fR       match a pattern
    \fB--last\fR \fILAST\fR          match a preceeding time period (5m/3h/2d/etc)
    \fB--process\fR \fIPROCESS\fR    match a source process
    \fB--protocol\fR \fIPROTOCOL\fR  match a protocol
    \fB--range\fR \fIRANGE\fR        match a time range <YYYYMMDDhhmm-YYYYMMDDhhmm>
    \fB--utc\fR                use UTC for range matching
    \fB--rdhost\fR \fIDHOST\fR       filter out a destination host
    \fB--rgrep\fR \fIPATTERN\fR      filter out a pattern
    \fB--rprocess\fR \fIPROCESS\fR   filter out a source process
    \fB--rprotocol\fR \fIPROTOCOL\fR match a protocol
    \fB--rshost\fR \fISHOST\fR       filter out a source host
    \fB--rsource\fR \fISOURCE\fR     filter out a log source
    \fB--shost\fR \fISHOST\fR        match a source host
    \fB--source\fR \fISOURCE\fR      match a log source

  
  output options:
    \fB--outlog \fIOUTLOG\fR      set the output file for standard log output
    \fB--label \fILABEL\fR        set label type for OUTLOG (fname|fpath)
    \fB--outjson \fIOUTJSON\fR    set the output file for JSON output


  ==== Available parsing modules: ====
  
  \fBciscoios        \fR: cisco ios parsing module
  \fBldjson          \fR: logdissect JSON parsing module
  \fBsyslog          \fR: syslog (standard timestamp) parsing module
  \fBsyslogiso       \fR: syslog (ISO timestamp) parsing module
  \fBsyslognohost    \fR: syslog (standard timestamp, no host) parsing module
  \fBtcpdump         \fR: tcpdump terminal output parsing module
  \fBwindowsrsyslog  \fR: windows rsyslog agent log parsing module

.SH EXAMPLES
logdissect --last 10m auth.log
.br
logdissect --last 1h --process systemd --process CRON messages
.br
logdissect -v --range 20160202020202-20170227213200 --label fpath messages debug apache2/error.log
.br
logdissect -s --outlog myaccess.log --grep 192.168.1.56 --last 30d --label fname /var/log/apache2/access.log /var/log/apache2/other_vhosts_access.log

.SH NOTES
1. metadata: logdissect uses file modification times to assign years to syslog date stamps. This allows it to parse logs that span more than one year without a problem. If you are copying log files, always use 'cp -p' (or 'cp --preserve=timestamps') and 'scp -p' to preserve original mtimes and other file metadata.

2. Re-parsing: If you are planning on parsing an output file back into logdissect at some point, using JSON is highly recommended. The JSON output module uses date stamps that include a year. Re-parsing a standard log output file will cause problems if the file has a different mtime than the original logs.

3. --range shortcuts: The range module will fill in your ranges with zeroes if they are shorter than 14 characters. If you want to get a range of 20170204120000 to 20170204130000, you can save time and use 2017020412 and 2017020413.
{'fraction_non_alphanumeric': 0.1033312115425419, 'fraction_numerical': 0.02249098238913643, 'mean_word_length': 2.5519969856819893, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '299610', 'n_tokens_mistral': 3549, 'n_tokens_neox': 3195, 'n_words': 1099}