<commit_before>.TH LOGDISSECT 3
.SH NAME
The logdissect module contains utilities for parsing, merging, filtering, and exporting syslog data.

.SH SYNOPSIS
    import logdissect

    # Parsing
    myparser = logdissect.parsers.<parser>.ParseModule()
    attribute_dict = myparser.parse_line('<RAW_LINE>')
    file_dict = myparser.parse_file('<PATH/TO/FILE>')

    # Filtering
    myfilter = logdissect.filters.<filter>.FilterModule()
    filterd_dict = myfilter.filter(file_dict, values=['<VALUE1>', '<VALUE2>'])

    # Output
    myoutput = logdissect.output.<output>.OutputModule()
    myoutput.write_output(data, filename='FILENAME')

    # Time Zones
    entry = logdissect.utils.convert_standard_datestamp(entry)
    entry = logdissect.utils.convert_nodate_datestamp(entry, datetimeobject)
    entry = logdissect.utils.convert_iso_datestamp(entry)
    entry = logdissect.utils.convert_unix_datestamp(entry)
    entry = logdissect.utils.convert_now_datestamp(entry)

    entry = logdissect.utils.get_utc_date(entry)

    # Merging
    log_dict = logdissect.utils.merge_logs(dataset, sort={True|False})

.SH DESCRIPTION
The logdissect module comes with the logdissect log analysis program. It contains objects which can be used to parse log lines and files, merge and filter logs, and output to a few formats.

.SH PARSER MODULES
\fBmyparser = logdissect.parsers.<parser>.ParseModule()\fR
.br
Replace <parser> with one of the available parsers:
    \fBciscoios\fR - Cisco IOS logs
    \fBsyslog\fR - standard syslog
    \fBsyslogiso\fR - syslog with ISO 8601 datestamp
    \fBsyslognohost\fR - syslog with no host attribute
    \fBtcpdump\fR - tcpdump terminal output
    \fBsojson\fR - logdissect single-object JSON output
    \fBlinejson\fR - logdissect object-per-line JSON output
    \fBwebaccess\fR - web access logs
    \fBwindowsrsyslog\fR - Windows rsyslog agent forwarded logs
    
Parsers have two methods (except the sojson parser, which has no parse_line() method):

\fBfile_dict = myparser.parse_file(\fI<file>\fB)\fR
.br
Accepts a filename as input, and returns a dictionary with some metadata, and a list of entry dictionaries (`entries`).

Parsers have a \fBtzone\fR attribute that uses standard ISO 8601 offset to UTC (e.g. \fB+0500\fR, \fB-0200\fR); if not set, logdissect will attempt to get current time zone data from the local system (unless a time zone is already present, such as in the syslogiso parser, or the sojson parser).

\fBentry_dict = myparser.parse_line(\fI<line>\fB)\fR
.br
Accepts a log line as input, and returns a dictionary of strings. There are two built-in keys, \fBraw_text\fR and \fBparser\fR, and parsers can add their own keys.

Parsers have a \fBdatestamp_type\fR attribute that defines how timestamps will be converted. The options are as follows:

    \fBstandard\fR - standard syslog date stamps
    \fBnodate\fR - time stamps with no date (i.e. tcpdump)
    \fBiso\fR - ISO8601 timestamps
    \fBwebaccess\fR - web access log date stamps
    \fBunix\fR - Unix timestamps
    \fBnow\fR - always set date stamp to time parsed
    \fBNone\fR - skip conversion

Conversion happens with any parser that has a \fBdate_stamp\fR field in \fBfields\fR (the \fBnow\fR datestamp type doesn't require a \fBdate_stamp\fR field), and adds the following attributes to the entry dictionary:

    \fByear\fR - a 4-digit string (or None)
    \fBmonth\fR - a 2-digit string
    \fBday\fR - a 2-digit string
    \fBtstamp\fR - a 6-digit string with optional decimal and extra places
    \fBtzone\fR - \fB+\fR or \fB-\fR followed by a 4-digit offset to utc (HHMM)
    \fBnumeric_date_stamp\fR - a datestamp in the form of YYYYmmddHHMMSS[.ffffff]
    \fBdate_stamp\fR - a standard date stamp (added for `now` datestamp type only)

The sojson parser has no parse_line() method.

.SH FILTER MODULES
\fBmyfilter = logdissect.filters.<filter>.FilterModule()\fR
.br
Replace <filter> with one of the available filters:

    \fBsource\fR - match a log source
    \fBrsource\fR - filter out a log source
    \fBrange\fR - match a time range
    \fBlast\fR - match a preceeding time range
    \fBgrep\fR - match entries containing a regular expression
    \fBrgrep\fR - filter out entries containing a regular expression
    \fBshost` - match a source host
    \fBrshost` - filter out a source host
    \fBdhost` - match a destination host
    \fBrdhost` - filter out a destination host
    \fBprocess` - match a source process
    \fBrprocess` - filter out a source process
    \fBprotocol` - match a protocol
    \fBrprotocol` - filter out a protocol

Filters have one method, \fBfilter_values\fR. Usage for all filters except \fBlast\fR and \fBrange\fR:

\fBnew_data = myfilter.filter_data(\fIdata\fR, \fIvalues=['VALUE1', 'VALUE2']\fB)\fR
.br
\fBdata\fR should be a log dictionary, with an \fBentries\fR value that contains a list of event dictionaries. \fBvalues\fR is a list containing strings to match or filter out.
.br
Syntax for the \fBlast\fR and \fBrange\fR filters differs slighty. Instead of \fBvalues\fR, they are passed \fBvalue\fR, which is a single string. The format of \fBvalue\fR:

    \fBrange\fR filter - YYYYmmddHHMMSS-YYYYmmddHHMMSS (time values can be shortened; filter will fill in \fB0\fRs)
    \fBlast\fR filter - a number, followed by either \fBm\fR for minutes, \fBh\fR for hours, or \fBd\fR for days (e.g. \fB20m\fR)

Time-based filters filter on the \fBnumeric_date_stamp\fR value. The \fBrange\fR filter also has a \fButc\fR keyword argument that defaults to false. If set to \fBTrue\fR, it will filter based on \fBnumeric_date_stamp_utc\fR.

.SH OUTPUT MODULES
\fBmyoutput = logdissect.output.<output>.OutputModule()\fR
.br
Replace <output> with one of the available filters:

    \fBlog\fR - outputs to standard log file format
    \fBsojson\fR - outputs entry list to a single json object
    \fBlinejson\fR - outputs one json entry dictionary object per line

Output modules have one method, \fBwrite_output\fR. Usage:

\fBmyoutput.write_output(\fIdata\fR, \fIfilename='FILENAME'\fB)\fR
.br
\fBdata\fR should be a log dictionary, with an \fBentries\fR value that contains a list of event dictionaries.<commit_msg>Update API docs<commit_after>.TH LOGDISSECT 3
.SH NAME
The logdissect module contains utilities for parsing, merging, filtering, and exporting syslog data.

.SH SYNOPSIS
    import logdissect

    # Parsing
    myparser = logdissect.parsers.<parser>.ParseModule()
    attribute_dict = myparser.parse_line('<RAW_LINE>')
    file_dict = myparser.parse_file('<PATH/TO/FILE>')

    # Filtering
    myfilter = logdissect.filters.<filter>.FilterModule()
    filterd_dict = myfilter.filter(file_dict, values=['<VALUE1>', '<VALUE2>'])

    # Output
    myoutput = logdissect.output.<output>.OutputModule()
    myoutput.write_output(data, filename='FILENAME')

    # Time Zones
    entry = logdissect.utils.convert_standard_datestamp(entry)
    entry = logdissect.utils.convert_nodate_datestamp(entry, datetimeobject)
    entry = logdissect.utils.convert_iso_datestamp(entry)
    entry = logdissect.utils.convert_unix_datestamp(entry)
    entry = logdissect.utils.convert_now_datestamp(entry)

    entry = logdissect.utils.get_utc_date(entry)

    # Merging
    log_dict = logdissect.utils.merge_logs(dataset, sort={True|False})

.SH DESCRIPTION
The logdissect module comes with the logdissect log analysis program. It contains objects which can be used to parse log lines and files, merge and filter logs, and output to a few formats.

.SH PARSER MODULES
\fBmyparser = logdissect.parsers.<parser>.ParseModule()\fR
.br
Replace <parser> with one of the available parsers:
    \fBciscoios\fR - Cisco IOS logs
    \fBsyslog\fR - standard syslog
    \fBsyslogiso\fR - syslog with ISO 8601 datestamp
    \fBsyslognohost\fR - syslog with no host attribute
    \fBtcpdump\fR - tcpdump terminal output
    \fBsojson\fR - logdissect single-object JSON output
    \fBlinejson\fR - logdissect object-per-line JSON output
    \fBwebaccess\fR - web access logs
    \fBwindowsrsyslog\fR - Windows rsyslog agent forwarded logs
    
Parsers have two methods (except the sojson parser, which has no parse_line() method):

\fBfile_dict = myparser.parse_file(\fI<file>\fB)\fR
.br
Accepts a filename as input, and returns a dictionary with some metadata, and a list of entry dictionaries (`entries`).

Parsers have a \fBtzone\fR attribute that uses standard ISO 8601 offset to UTC (e.g. \fB+0500\fR, \fB-0200\fR); if not set, logdissect will attempt to get current time zone data from the local system (unless a time zone is already present, such as in the syslogiso parser, or the sojson parser).

\fBentry_dict = myparser.parse_line(\fI<line>\fB)\fR
.br
Accepts a log line as input, and returns a dictionary of strings. There are two built-in keys, \fBraw_text\fR and \fBparser\fR, and parsers can add their own keys.

Parsers have a \fBdatestamp_type\fR attribute that defines how timestamps will be converted. The options are as follows:

    \fBstandard\fR - standard syslog date stamps
    \fBnodate\fR - time stamps with no date (i.e. tcpdump)
    \fBiso\fR - ISO8601 timestamps
    \fBwebaccess\fR - web access log date stamps
    \fBunix\fR - Unix timestamps
    \fBnow\fR - always set date stamp to time parsed
    \fBNone\fR - skip conversion

Conversion happens with any parser that has a \fBdate_stamp\fR field in \fBfields\fR (the \fBnow\fR datestamp type doesn't require a \fBdate_stamp\fR field), and adds the following attributes to the entry dictionary:

    \fByear\fR - a 4-digit string (or None)
    \fBmonth\fR - a 2-digit string
    \fBday\fR - a 2-digit string
    \fBtstamp\fR - a 6-digit string with optional decimal and extra places
    \fBtzone\fR - \fB+\fR or \fB-\fR followed by a 4-digit offset to utc (HHMM)
    \fBnumeric_date_stamp\fR - a datestamp in the form of YYYYmmddHHMMSS[.ffffff]
    \fBdate_stamp\fR - a standard date stamp (added for `now` datestamp type only)

The sojson parser has no parse_line() method.

.SH FILTER MODULES
\fBmyfilter = logdissect.filters.<filter>.FilterModule()\fR
.br
Replace <filter> with one of the available filters:

    \fBsource\fR - match a log source
    \fBrsource\fR - filter out a log source
    \fBrange\fR - match a time range
    \fBlast\fR - match a preceeding time range
    \fBgrep\fR - match entries containing a regular expression
    \fBrgrep\fR - filter out entries containing a regular expression
    \fBshost` - match a source host
    \fBrshost` - filter out a source host
    \fBdhost` - match a destination host
    \fBrdhost` - filter out a destination host
    \fBprocess` - match a source process
    \fBrprocess` - filter out a source process
    \fBprotocol` - match a protocol
    \fBrprotocol` - filter out a protocol

Filters have one method, \fBfilter_values\fR. Usage for all filters except \fBlast\fR and \fBrange\fR:

\fBnew_data = myfilter.filter_data(\fIdata\fR, \fIvalues=['VALUE1', 'VALUE2']\fB)\fR
.br
\fBdata\fR should be a log dictionary, with an \fBentries\fR value that contains a list of event dictionaries. \fBvalues\fR is a list containing strings to match or filter out.

Syntax for the \fBlast\fR and \fBrange\fR filters differs slighty. Instead of \fBvalues\fR, they are passed \fBvalue\fR, which is a single string. The format of \fBvalue\fR:

    \fBrange\fR filter - YYYYmmddHHMMSS-YYYYmmddHHMMSS (time values can be shortened; filter will fill in \fB0\fRs)
    \fBlast\fR filter - a number, followed by either \fBm\fR for minutes, \fBh\fR for hours, or \fBd\fR for days (e.g. \fB20m\fR)

Time-based filters filter on the \fBnumeric_date_stamp\fR value. The \fBrange\fR filter also has a \fButc\fR keyword argument that defaults to false. If set to \fBTrue\fR, it will filter based on \fBnumeric_date_stamp_utc\fR.

.SH OUTPUT MODULES
\fBmyoutput = logdissect.output.<output>.OutputModule()\fR
.br
Replace <output> with one of the available filters:

    \fBlog\fR - outputs to standard log file format
    \fBsojson\fR - outputs entry list to a single json object
    \fBlinejson\fR - outputs one json entry dictionary object per line

Output modules have one method, \fBwrite_output\fR. Usage:

\fBmyoutput.write_output(\fIdata\fR, \fIfilename='FILENAME'\fB)\fR
.br
\fBdata\fR should be a log dictionary, with an \fBentries\fR value that contains a list of event dictionaries.
{'fraction_non_alphanumeric': 0.08528387462541508, 'fraction_numerical': 0.005345428039199806, 'mean_word_length': 3.5330396475770924, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7273880', 'n_tokens_mistral': 4281, 'n_tokens_neox': 4007, 'n_words': 1408}