<commit_before># Sieve Bot

The sieve bot is used to filter and/or modify bots based on a set of rules. The
rules are specified in an external configuration file and with a syntax similar
to the [Sieve language](http://sieve.info/) used for mail filtering.

Each rule defines a set of matching conditions on received events. Events can be
matched based on keys and values in the event. If the processed event matches a
rule's conditions, the corresponding actions are performed. Actions can specify
whether the event should be kept or dropped in the pipeline (filtering actions)
or if keys and values should be changed (modification actions).


## Examples

The following excerpts illustrate some of the basic features of the sieve file
format:

```
if :exists source.fqdn {
  keep  // aborts processing of subsequent rules and forwards the event.
}


if :notexists source.abuse_contact || source.abuse_contact =~ '.*<EMAIL>' {
  drop  // aborts processing of subsequent rules and drops the event.
}


if classification.type == ['phishing', 'malware'] && source.fqdn =~ '.*\.(ch|li)$' {
  add comment = 'domainabuse'
  keep
} elsif classification.type == 'scanner' {
  add comment = 'ignore'
  drop
} else {
  remove comment
}
```


## Parameters

The sieve bot only takes one parameter:
 * `file` - filesystem path the the sieve file


## Reference

### Sieve File Structure

The sieve file contains an arbitrary number of rules of the form:

```
if EXPRESSION {
    ACTIONS
} elif EXPRESSION {
    ACTIONS
} else {
    ACTIONS
}
```


###  Expressions

Each rule specifies on or more expressions to match an event based on its keys
and values. Event keys are specified as strings without quotes. String values
must be enclosed in single quotes. Numeric values can be specified as integers
or floats and are unquoted. Following operators may be used to match events:

 * `:exists` and `:notexists` match if a given key exists, for example:

    ```if :exists source.fqdn { ... }```

 * `==` and `!=` match for equality of strings and numbers, for example:

   ```if feed.name != 'acme-security' || feed.accuracy == 100 { ... }```

 * `:contains` matches on substrings.

 * `=~` matches strings based on the given regex. `!~` is the inverse regex
 match.

 * Numerical comparisons are evaluated with `<`, `<=`, `>`, `>=`.

 * Values to match against can also be specified as list, in which case any one
 of the values will result in a match:

   ```if source.ip == ['8.8.8.8', '8.8.4.4'] { ... }```

  In this case, the event will match if it contains a key `source.ip` with
  either value `8.8.8.8` or `8.8.4.4`.


### Actions

If part of a rule matches the given conditions, the actions enclosed in `{` and
`}` are applied. By default, all events that are matched or not matched by rules
in the sieve file will be forwarded to the next bot in the pipeline, unless the
`drop` action is applied.

 * `add` adds a key value pair to the event. This action only applies if the key
 is not yet defined in the event. If the key is already defined, the action is
 ignored. Example:

   ```add comment = 'hello, world'```

 * `add!` same as above, but will force overwrite the key in the event.

 * `modify` modifies an existing value for a key. Only applies if the key is
already defined. If the key is not defined in the event, this action is ignored.
Example:

   ```modify feed.accuary = 50```

 * `remove` removes a key/value from the event. Action is ignored if the key is
 not defined in the event. Example:

    ```remove extra.comments```

 * `keep` marks the event to be forwarded to the next bot in the pipeline
 (same as the default behaviour), but in addition the sieve file processing is
 interrupted upon reaching this action.

 * `drop` marks the event to be dropped. The event will not be forwarded to the
 next bot in the pipeline. The sieve file processing is interrupted upon
 reaching this action. No other actions may be specified besides the `drop`
 action within `{` and `}`.<commit_msg>Added section about comments, antoinet/intelmq#9<commit_after># Sieve Bot

The sieve bot is used to filter and/or modify bots based on a set of rules. The
rules are specified in an external configuration file and with a syntax similar
to the [Sieve language](http://sieve.info/) used for mail filtering.

Each rule defines a set of matching conditions on received events. Events can be
matched based on keys and values in the event. If the processed event matches a
rule's conditions, the corresponding actions are performed. Actions can specify
whether the event should be kept or dropped in the pipeline (filtering actions)
or if keys and values should be changed (modification actions).


## Examples

The following excerpts illustrate some of the basic features of the sieve file
format:

```
if :exists source.fqdn {
  keep  // aborts processing of subsequent rules and forwards the event.
}


if :notexists source.abuse_contact || source.abuse_contact =~ '.*<EMAIL>' {
  drop  // aborts processing of subsequent rules and drops the event.
}


if classification.type == ['phishing', 'malware'] && source.fqdn =~ '.*\.(ch|li)$' {
  add comment = 'domainabuse'
  keep
} elsif classification.type == 'scanner' {
  add comment = 'ignore'
  drop
} else {
  remove comment
}
```


## Parameters

The sieve bot only takes one parameter:
 * `file` - filesystem path the the sieve file


## Reference

### Sieve File Structure

The sieve file contains an arbitrary number of rules of the form:

```
if EXPRESSION {
    ACTIONS
} elif EXPRESSION {
    ACTIONS
} else {
    ACTIONS
}
```


###  Expressions

Each rule specifies on or more expressions to match an event based on its keys
and values. Event keys are specified as strings without quotes. String values
must be enclosed in single quotes. Numeric values can be specified as integers
or floats and are unquoted. Following operators may be used to match events:

 * `:exists` and `:notexists` match if a given key exists, for example:

    ```if :exists source.fqdn { ... }```

 * `==` and `!=` match for equality of strings and numbers, for example:

   ```if feed.name != 'acme-security' || feed.accuracy == 100 { ... }```

 * `:contains` matches on substrings.

 * `=~` matches strings based on the given regex. `!~` is the inverse regex
 match.

 * Numerical comparisons are evaluated with `<`, `<=`, `>`, `>=`.

 * Values to match against can also be specified as list, in which case any one
 of the values will result in a match:

   ```if source.ip == ['8.8.8.8', '8.8.4.4'] { ... }```

  In this case, the event will match if it contains a key `source.ip` with
  either value `8.8.8.8` or `8.8.4.4`.


### Actions

If part of a rule matches the given conditions, the actions enclosed in `{` and
`}` are applied. By default, all events that are matched or not matched by rules
in the sieve file will be forwarded to the next bot in the pipeline, unless the
`drop` action is applied.

 * `add` adds a key value pair to the event. This action only applies if the key
 is not yet defined in the event. If the key is already defined, the action is
 ignored. Example:

   ```add comment = 'hello, world'```

 * `add!` same as above, but will force overwrite the key in the event.

 * `modify` modifies an existing value for a key. Only applies if the key is
already defined. If the key is not defined in the event, this action is ignored.
Example:

   ```modify feed.accuary = 50```

 * `remove` removes a key/value from the event. Action is ignored if the key is
 not defined in the event. Example:

    ```remove extra.comments```

 * `keep` marks the event to be forwarded to the next bot in the pipeline
 (same as the default behaviour), but in addition the sieve file processing is
 interrupted upon reaching this action.

 * `drop` marks the event to be dropped. The event will not be forwarded to the
 next bot in the pipeline. The sieve file processing is interrupted upon
 reaching this action. No other actions may be specified besides the `drop`
 action within `{` and `}`.


 ### Comments

 Comments may be used in the sieve file: all characters after `//` and until the end of the line will be ignored.
{'fraction_non_alphanumeric': 0.09004159530217763, 'fraction_numerical': 0.005260582334230487, 'mean_word_length': 3.6396140749148693, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 8, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3294489', 'n_tokens_mistral': 2334, 'n_tokens_neox': 2200, 'n_words': 1194}