<commit_before># lumberjack

o/~ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! o/~

Collect logs locally in preparation for processing elsewhere!

Problem: logstash jar releases are too fat for constrained systems.

Goal: Something small, fast, and light-weight to ship local logs externally.

## Requirements

* minimal resource usage
* configurable event data
* encryption and compression

Simple inputs only:

* follow files, respect rename/truncation conditions
* local sockets, maybe, if syslog(3) is worth supporting.
* stdin, useful for things like 'varnishlog | lumberjack ...'

Simple outputs only:

* custom wire event protocol (TBD)

## Tentative idea:

    # Ship apache logs in real time to somehost:12345
    ./lumberjack --target somehost:12345 /var/log/apache/access.log ...

    # Ship apache logs with additional log fields:
    ./lumberjack --target foo:12345 --field host=$HOSTNAME --field role=apt-repo /mnt/apt/access.log

* Serialization: msgpack (likely)
* Encryption: SSL
* Authentication (both directions): SSL certificates
* Compression: TLS v1 comes with compression, might be sufficient.<commit_msg>update readme to better reflect reality<commit_after># lumberjack

o/~ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! o/~

Collect logs locally in preparation for processing elsewhere!

Problem: logstash jar releases are too fat for constrained systems.

## Goal s

* minimize resource usage where possible (cpu, memory, network)
* secure transmission of logs
* configurable event data
* easy to deploy with minimal moving parts.

Simple inputs only:

* follow files, respect rename/truncation conditions
* stdin, useful for things like 'varnishlog | lumberjack ...'

## Implementation details 

Below is valid as of 2012/09/19

### Minimize resource usage

* sets small resource limits (memory, open files) on start up based on the number of files being watched
* cpu: sleeps when there is nothing to do
* network/cpu: sleeps if there's a network failure
* network: uses zlib for compression

### secure transmission

* uses openssl to transport logs. Currently supports verifying the server
  certificate only (so you know who you are sending to).

### configurable event data

* the protocol lumberjack uses supports sending a string:string map
* the lumberjack tool lets you specify arbitrary extra data with `--field name=value`
{'fraction_non_alphanumeric': 0.07154213036565978, 'fraction_numerical': 0.009936406995230525, 'mean_word_length': 4.321353065539112, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '581173', 'n_tokens_mistral': 707, 'n_tokens_neox': 648, 'n_words': 336}