<commit_before># evd

A local event and metrics collection daemon prototype.

EVD is an event agent prototype meant to run on a single host and receive
metrics and events using a set of standard protocols.

It acts as an abstraction layer between between a service and monitor system,
it is capable of rich metadata decoration of the events passing through it.

* [Usage](#usage)
* [Events and Metrics](#events-and-metrics)
* [Debugging](#debugging)
* [Tunneling and multi-tenancy](#tunneling-and-multi-tenancy)
  * [Description](#description)
  * [Comparison to other tunneling solutions](#comparison-to-other-tunneling-solutions)
* [Terminology](#terminology)

## Usage

The simplest possible use is to install all dependencies and run it using the
supplied example configuration.

```
bundle install
bin/evd -c docs/simple.conf
```

This will start up an instance that periodically reports statistics about
itself.

You can try out it's capabilities by running the supplied
[docs/client-test.rb](docs/client-test.rb).

```
ruby docs/client-test.rb
```

If you want to experiment with multitenancy, use the provided
[docs/tunnel.conf](docs/tunnel.conf) as base.
For the example [docs/client-test.rb](docs/client-test.rb) to work, you have to
start a tunneling agent.

You can do this by running the provided [bin/tunnel-agent](bin/tunnel-agent)
which is a reference implementation of the [tunneling
protocol](#tunneling-and-multi-tenancy).

```
$ bin/tunnel-agent
INFO:__main__:connected
INFO:__main__:CONFIG: {...}
...
```

It should now be possible to use the provided
[docs/client-test.rb](docs/client-test.rb) the same way as you did before.

## Events and Metrics

The two types of data that EVD processes are *events* and *metrics*.

An input or output plugin can both produce and consume *events* and *metrics*,
but they are treated slightly differently by *Core*.

* **events** Are passed as-is, without being processed.
* **metrics** Can be optionally processed, but default to being passed as-is.

*Core* is also responsible for *decorating* both events and metrics with
*metadata* by adding *tags* and *attributes*.

*An output plugin does not* have to support forwarding this *metadata* and in
fact most do not.
Instead it is up to the system administrator to choose a suitable output
scheme.

*Core* makes a distinction between an **input event/metric** and an **output
event/metric**.

An **input** type of data is just a Hash with keyword fields and values.
It is designed like this to allow for input plugins to provide *terse*
information like only the *:key* and *:value* fields of a metric (see [the
statsd plugin](lib/evd/plugin/statsd.rb) for a good example).

An **output** type of data is in contrast defined by a Struct.
This is because we want the metrics and events to be *consistent* after they
have passed through *Core* and reaches an output plugin.

For the schemas of both input and output data types, see
[lib/evd/event.rb](lib/evd/event.rb) and
[lib/evd/metric.rb](lib/evd/metric.rb).

## Debugging

While the agent is running, it is possible to sniff all internal input and
output channels by enabling the debug component.

This is done by adding the following to your evd configuration.

```
:debug: {}
```

This will setup the component to listen on the default debug port (9999).

The traffic can now be sniffed with a tool like netcat.

```
$ nc localhost 9999
{"id":"core.output","type":"event","data": ...}
{"id":"tunnel.input/127.0.0.1:55606","type":"event","data": ...}
{"id":"core.output","type":"event","data": ...}
{"id":"tunnel.input/127.0.0.1:55606","type":"event","data": ...}
{"id":"tunnel.input/127.0.0.1:55606","type":"metric","data": ...}
...
```

Each line consists of a JSON object with the following fields.

**id** The id that the specified events can be grouped by, this indicates
which channel the traffic was sniffed of.

**type** The type of the *data* field.

**data** Data describing the sniffed event according to specified *type*.

## Tunneling and multi-tenancy

Multi-tenancy is the act of supporting multiple distinct client services with
unique metadata (host, tags, attributes) on a single EVD agent.

This is currently achieved with the *tunnel* plugin.

**The tunneling protocol is experimental and will be subject to future change**

### Description

In multi-tenancy we distinguish betweeh the *host* and the *guest* system.

In EVD, the *host* system runs the EVD agent, and every *guest* runs a small
[*tunneling agent*](bin/tunnel-agent) which connects to the *host*.

The tunneling agent is responsible for doing the following.

* Send metadata (host, tags, attributes) about the *guest*.
* Proxy level 4 (TCP, UDP) connection to the *host* agent.
* Receive configuration from the *host* agent of what needs proxying and
  reconfigure itself accordingly.

On the *host* the tunnel is an *input* plugin called *tunnel* which accepts
connections from its *guests*.

### Protocol

```<commit_msg>A few (incomplete) documentation changes<commit_after># evd

A highly flexible, multi-protocol events and metrics forwarder capable
of merging and decorating events and metrics from a number of sources
and multiplexing them to number of event and metric consumers.

EVD is a deamon meant to run on a single host and receive metrics and events
using a set of standard protocols.


**TODO (parmus): Redo this:**
> It acts as an abstraction layer between between a service and monitor system,
> it is capable of rich metadata decoration of the events passing through it.


EVD is currently a prototype implemented to explore this design. Your milage might vary.

* [Usage](#usage)
* [EVD vs. collectd](#evd-vs.-collectd)
* [Events and Metrics](#events-and-metrics)
* [Debugging](#debugging)
* [Tunneling and multi-tenancy](#tunneling-and-multi-tenancy)
  * [Description](#description)
  * [Comparison to other tunneling solutions](#comparison-to-other-tunneling-solutions)
* [Terminology](#terminology)

## Usage

The simplest possible use is to install all dependencies and run it using the
supplied example configuration.

```bash
bundle install
bin/evd -c docs/simple.conf
```

This will start up an instance that periodically reports statistics about
itself.

You can now send events and metrics to it using one of the enabled input protocols,
e.g. the carbon protocol or the JSON-line protocol:

```bash
echo "local.random.diceroll 4  `date +%s`" | nc -q0 localhost 2003
echo '{"type": "metric", "key": "local.random.diceroll", "value": 6}' | nc -q0 localhost 3000
```

You can try out more advanced protocols using the supplied sample client:
[docs/client-test.rb](docs/client-test.rb).

```bash
ruby docs/client-test.rb
```

If you have the log output plugin enabled, you should see these metrics written to the log.






**TODO (parmus): Consider moving this section**
> If you want to experiment with multitenancy, use the provided
> [docs/tunnel.conf](docs/tunnel.conf) as base.
> For the example [docs/client-test.rb](docs/client-test.rb) to work, you have to
> start a tunneling agent.

> You can do this by running the provided [bin/tunnel-agent](bin/tunnel-agent)
> which is a reference implementation of the [tunneling
> protocol](#tunneling-and-multi-tenancy).

```
$ bin/tunnel-agent
INFO:__main__:connected
INFO:__main__:CONFIG: {...}
...
```

> It should now be possible to use the provided
> [docs/client-test.rb](docs/client-test.rb) the same way as you did before.






## EVD vs. collectd

EVD is inspired by the pluggable infrastructure provided by [collectd](http://collectd.org/), but is designed
for a slightly different use-case and using different design principals.

* collectd does metric collection inside it's core using plugins, i.e. if you want
  to add and collect some custom metric, then you write a new collectd plugin or
  extension. In contrast, EVD only deals with forwarding metrics sent through it and
  does NOT metric collection what so ever. Any actual collection must happen outside EVD,
  and EVD only provides the transport (forwarding) layer.
* collectd is optimized for performance, scalability and portability. It's written in C
  and it makes heavily use of internal buffers and threads to deliver this performance.
  While this makes collectd very fast, it also make it very hard to debug, when something
  goes wrong. In contract, EVD is written in Ruby with introspection and debuggability in
  mind, it's core is completely event-driven and synchronous (single-process and
  single-threaded) and does no internal buffering. This makes EVD very simple to debug and
  understand albeit not as performant as collectd. However we believe EVD's simple structure
  makes it performant *enough*.
* collectd tries very hard to deliver reliable metrics. collectd will buffer (both in memory
  and ultimately on disc) and resent metrics, if it has trouble delivering them to it's
  consumers, and it will safe unsent metrics to and reload from disc on shutdown and restart.
  All in all, collectd behaves very much like TCP, trying to be very reliable and treating
  every measured metrics as precious pieces of data.
  In contract EVD behaves more like UDP, will do no buffering or saving to disc, and will
  happily drop metrics, if it has trouble delivering them. This makes EVD very resilient to
  e.g. high CPU load, slow network and work overload (to many or to fast incoming metrics).
  EVD will generate internal events, when dropping traffic to notify consumers about this
  behavior.


**TODO (parmus): Could we put collectd in front of EVD for collection?**

## Writing plugins

**TODO (parmus)**

## Events and Metrics

The two types of data that EVD processes are *events* and *metrics*.


**TODO (parmus): This makes no sense:**
> An input or output plugin can both produce and consume *events* and *metrics*,
> but they are treated slightly differently by *Core*.

* **events** Are passed as-is, without being processed.
* **metrics** Can be optionally processed, but default to being passed as-is.

*Core* is also responsible for *decorating* both events and metrics with
*metadata* by adding *tags* and *attributes*.

*An output plugin does not* have to support forwarding this *metadata* and in
fact most do not.
Instead it is up to the system administrator to choose a suitable output
scheme.

*Core* makes a distinction between an **input event/metric** and an **output
event/metric**.

An **input** type of data is just a Hash with keyword fields and values.
It is designed like this to allow for input plugins to provide *terse*
information like only the *:key* and *:value* fields of a metric (see [the
statsd plugin](lib/evd/plugin/statsd.rb) for a good example).

An **output** type of data is in contrast defined by a Struct.
This is because we want the metrics and events to be *consistent* after they
have passed through *Core* and reaches an output plugin.

For the schemas of both input and output data types, see
[lib/evd/event.rb](lib/evd/event.rb) and
[lib/evd/metric.rb](lib/evd/metric.rb).

## Debugging

While the agent is running, it is possible to sniff all internal input and
output channels by enabling the debug component.

This is done by adding the following to your evd configuration.

```
:debug: {}
```

This will setup the component to listen on the default debug port (9999).

The traffic can now be sniffed with a tool like netcat.

```
$ nc localhost 9999
{"id":"core.output","type":"event","data": ...}
{"id":"tunnel.input/127.0.0.1:55606","type":"event","data": ...}
{"id":"core.output","type":"event","data": ...}
{"id":"tunnel.input/127.0.0.1:55606","type":"event","data": ...}
{"id":"tunnel.input/127.0.0.1:55606","type":"metric","data": ...}
...
```

Each line consists of a JSON object with the following fields.

**id** The id that the specified events can be grouped by, this indicates
which channel the traffic was sniffed of.

**type** The type of the *data* field.

**data** Data describing the sniffed event according to specified *type*.

## Tunneling and multi-tenancy

**TODO(parmus): Rewrite with focus on the container use-case **

Multi-tenancy is the act of supporting multiple distinct client services with
unique metadata (host, tags, attributes) on a single EVD agent.

This is currently achieved with the *tunnel* plugin.

**The tunneling protocol is experimental and will be subject to future change**

### Description

In multi-tenancy we distinguish betweeh the *host* and the *guest* system.

In EVD, the *host* system runs the EVD agent, and every *guest* runs a small
[*tunneling agent*](bin/tunnel-agent) which connects to the *host*.

The tunneling agent is responsible for doing the following.

* Send metadata (host, tags, attributes) about the *guest*.
* Proxy level 4 (TCP, UDP) connection to the *host* agent.
* Receive configuration from the *host* agent of what needs proxying and
  reconfigure itself accordingly.

On the *host* the tunnel is an *input* plugin called *tunnel* which accepts
connections from its *guests*.

### Protocol

```
{'fraction_non_alphanumeric': 0.09632922062993333, 'fraction_numerical': 0.0073568855851023065, 'mean_word_length': 4.46940486169321, 'pattern_counts': {'":': 33, '<': 3, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5743973', 'n_tokens_mistral': 3780, 'n_tokens_neox': 3591, 'n_words': 1759}