<commit_before>.\" This document is in the public domain.
.\" vim: fdm=marker
.
.\" FRONT MATTER {{{
.Dd Oct 10, 2012
.Os
.Dt LOGDEMUX 1
.
.Sh NAME
.Nm logdemux
.Nd Logfile demultiplexer
.\" FRONT MATTER }}}
.
.\" SYNOPSIS {{{
.Sh SYNOPSIS
.Nm
.Ar rules
.Ar prefix
.\" SYNOPSIS }}}
.
.\" DESCRIPTION {{{
.Sh DESCRIPTION
.Nm
stores lines read from the standard input into different
files based on matching those lines with user-specified
regular expressions.
.
.Bl -tag -width "xx"
.It Ar rules
configuration file
.It Ar prefix
substituted into
.Li sink
values
.Pq Li \&%P
.El
.Pp
.
.Ss Rules
.
The filtering rules are given in an INI-formatted file, see
Iniphile for details on general syntax.
.Pp
Configuration must contain a
.Ql rules
section, with
.Ql order
property.
Its value is expected to list rule section names in the order they
should be used.
.Pp
Each rule section defines a rule with properties
.Ql match ,
.Ql sink ,
and optionally
.Ql final .
.Pp
.
.Bl -tag -width "xx"
.It Li match
Perl-compatible regular expression.
.
.It Li sink
Log file path.
.Li \&%P
is replaced with the
.Ar prefix
argument.
.Li \&%D
is replaced with current date in the
.Li YYYY-MM-DD
format.
.
.It Li final
Boolean.
.Li true , yes , on , 1
are true, anything else is false.
.El
.
.Pp
If the present line does not match the pattern in
.Ql match ,
.Nm
moves on to the next rule as defined by
.Ql rules.order .
If the present line does match the regular expression from
.Ql match ,
it is written into the filename given in
.Ql sink .
Then, if the
.Ql final
property is present and true, processing of this line ends.
Otherwise, the next rule is tried.
.<commit_msg>manpage: better description of runtime behavior<commit_after>.\" This document is in the public domain.
.\" vim: fdm=marker
.
.\" FRONT MATTER {{{
.Dd Oct 10, 2012
.Os
.Dt LOGDEMUX 1
.
.Sh NAME
.Nm logdemux
.Nd Logfile demultiplexer
.\" FRONT MATTER }}}
.
.\" SYNOPSIS {{{
.Sh SYNOPSIS
.Nm
.Ar rules
.Ar prefix
.\" SYNOPSIS }}}
.
.\" DESCRIPTION {{{
.Sh DESCRIPTION
.Nm
stores lines read from the standard input into different
files based on matching those lines with user-specified
regular expressions.
.
.Bl -tag -width "xx"
.It Ar rules
configuration file
.It Ar prefix
substituted into
.Li sink
values
.Pq Li \&%P
.El
.Pp
.
.Ss Rules
.
The filtering rules are given in an INI-formatted file, see
Iniphile for details on general syntax.
.Pp
Configuration must contain a
.Ql rules
section, with
.Ql order
property.
Its value is expected to list rule section names in the order they
should be used.
.Pp
Each rule section defines a rule with properties
.Ql match ,
.Ql sink ,
and optionally
.Ql final .
.Pp
.
.Bl -tag -width "xx"
.It Li match
Perl-compatible regular expression.
.
.It Li sink
Log file path.
.Li \&%P
is replaced with the
.Ar prefix
argument.
.Li \&%D
is replaced with current date in the
.Li YYYY-MM-DD
format.
.
.It Li final
Boolean.
.Li true , yes , on , 1
are true, anything else is false.
.El
.
.Ss Operation
Runtime behavior conforms to this pseudocode:
.Bd -literal
  for each line of input
    for each rule of rules.order
      if rule.match line
        write line to rule.sink
        if rule.final
          next line
.Ed
.
{'fraction_non_alphanumeric': 0.09210115516703091, 'fraction_numerical': 0.004995316890415236, 'mean_word_length': 3.005, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5575865', 'n_tokens_mistral': 1147, 'n_tokens_neox': 1089, 'n_words': 505}