<commit_before>
![Flowgger](https://raw.github.com/jedisct1/flowgger/master/flowgger.png)

Flowgger
========

Flowgger is a fast, simple and lightweight data collector written in Rust.

It reads log entries over a given protocol, decodes them using a given
format, reencodes them into a different format, and asynchronously pushes the
result into a remote data store.

While not providing the same set of features and flexibility as tools
such as [Fluentd](http://www.fluentd.org/) or Logstash, it is orders
of magnitude faster and doesn't require a JVM.

Compilation and installation
----------------------------

The current code is written for rust-nightly (1.4.0), although it can work on
rust-stable with minor changes.

In addition to rust-nightly, the openssl-dev system package (or LibreSSL) is
required for TLS support.

After having intalled rust-nightly, compile with the usual:

    cargo build --release

And copy the `target/release/flowgger` file anywhere; this is the only file
you need.

Configuration
-------------

Flowgger reads its configuration from a file given as an argument:

    flowgger flowgger.toml

The configuration file consists of two main sections: `[input]` and `[output]`.

Input section
-------------

```toml
[input]
type = "syslog-tls"
listen = "0.0.0.0:6514"
format = "rfc5424"
tls_cert = "flowgger.pem"
tls_key = "flowgger.pem"
tls_ciphers = "ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256"
queuesize = 1000000
```

The currently supported values for the input `type` are `syslog-tcp` (text-based
syslog messages over a TCP socket) and `syslog-tls` (text-based syslog messages
over TLS).

When using TLS, `tls_ciphers` is optional and defaults to a safe suite, but
`tls_cert` and `tls_key` are required.

Up to `queuesize` messages may be buffered in memory if the final datastore
cannot keep up with the input rate.

The currently supported input `format` types are `rfc5424`, `gelf` and `ltsv`:

* [RFC 5424](https://tools.ietf.org/html/rfc5424),
* [GELF](https://www.graylog.org/resources/gelf-2/)
* [LTSV](http://ltsv.org)

### RFC5424

Record example:

    <23>1 2015-08-05T15:53:45.637824Z testhostname appname 69 42 [origin@123 software="test script" swVersion="0.0.1"] test message

RFC 5424 messages are assumed to be on a single line and to be made of valid
UTF8 sequences.<commit_msg>TLS 1.2 is required<commit_after>
![Flowgger](https://raw.github.com/jedisct1/flowgger/master/flowgger.png)

Flowgger
========

Flowgger is a fast, simple and lightweight data collector written in Rust.

It reads log entries over a given protocol, decodes them using a given
format, reencodes them into a different format, and asynchronously pushes the
result into a remote data store.

While not providing the same set of features and flexibility as tools
such as [Fluentd](http://www.fluentd.org/) or Logstash, it is orders
of magnitude faster and doesn't require a JVM.

Compilation and installation
----------------------------

The current code is written for rust-nightly (1.4.0), although it can work on
rust-stable with minor changes.

In addition to rust-nightly, the openssl-dev system package (or LibreSSL) is
required for TLS support.

After having intalled rust-nightly, compile with the usual:

    cargo build --release

And copy the `target/release/flowgger` file anywhere; this is the only file
you need.

Configuration
-------------

Flowgger reads its configuration from a file given as an argument:

    flowgger flowgger.toml

The configuration file consists of two main sections: `[input]` and `[output]`.

Input section
-------------

```toml
[input]
type = "syslog-tls"
listen = "0.0.0.0:6514"
format = "rfc5424"
tls_cert = "flowgger.pem"
tls_key = "flowgger.pem"
tls_ciphers = "ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256"
queuesize = 1000000
```

The currently supported values for the input `type` are `syslog-tcp` (text-based
syslog messages over a TCP socket) and `syslog-tls` (text-based syslog messages
over TLS).

When using TLS, `tls_ciphers` is optional and defaults to a safe suite, but
`tls_cert` and `tls_key` are required. TLS 1.2 is required.

Up to `queuesize` messages may be buffered in memory if the final datastore
cannot keep up with the input rate.

The currently supported input `format` types are `rfc5424`, `gelf` and `ltsv`:

* [RFC 5424](https://tools.ietf.org/html/rfc5424),
* [GELF](https://www.graylog.org/resources/gelf-2/)
* [LTSV](http://ltsv.org)

### RFC5424

Record example:

    <23>1 2015-08-05T15:53:45.637824Z testhostname appname 69 42 [origin@123 software="test script" swVersion="0.0.1"] test message

RFC 5424 messages are assumed to be on a single line and to be made of valid
UTF8 sequences.
{'fraction_non_alphanumeric': 0.11676202585293495, 'fraction_numerical': 0.03899131171858444, 'mean_word_length': 3.9841605068637804, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 6, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5053710', 'n_tokens_mistral': 1649, 'n_tokens_neox': 1475, 'n_words': 580}