<commit_before># Gollum

Gollum is a n:m log multiplexer that gathers messages from different sources and broadcasts them to a set of listeners.

There are a few basic terms used throughout Gollum:

* A "consumer" is a plugin that reads from an external source
* A "producer" is a plugin that writes to an external source
* A "stream" is a message channel between consumer(s) and producer(s)
* A "formatter" is a plugin that adds information to a message
* A "distributor" is a plugin that routes/filters messages on a given stream

Writing a custom plugin does not require you to change any additional code besides your new plugin file.

## Consumers (reading data)

* `Console` read from stdin.
* `File` read from a file (like tail).
* `Kafka` read from a [Kafka](http://kafka.apache.org/) topic.
* `Socket` read from a socket (gollum specfic protocol).

## Producers (writing data)

* `Console` write to stdin or stdout.
* `ElasticSearch` write to [elasticsearch](http://www.elasticsearch.org/) via http/bulk.
* `File` write to a file. Supports log rotation and compression.
* `Kafka` write to a [Kafka](http://kafka.apache.org/) topic.
* `Null` like /dev/null.
* `Facebook Scribe` send messages to a [scribe](https://github.com/facebookarchive/scribe) server.
* `Socket` send messages to a socket (gollum specfic protocol).

## Formatters (modifying data)

TODO

## Distributors (multiplexing)

* `Broadcast` send to all producers in a stream.
* `Random` send to a random roducers in a stream.
* `RoundRobin` switch the producer after each send in a round robin fashion.

## Installation

### From source

Installation from source requires the installation of the [Go toolchain](http://golang.org/).

```
$ go get .
$ go build .
$ gollum --help
```

## Usage

The simplest usage is to make a local profiler run with a predefined consiguration:

```
$ gollum -c gollum_profile.conf
```

This configuration generates random messages (via *consumer.Profiler*) and write this to a file called *log_profile.log* with file rotation after 512MB per file (via *producer.File*).

## Configuration

Configuration files are written in the YAML format and have to be loaded via command line switch.
Each plugin has a different set of configuration options which are currently described in the plugin itself, i.e. you can find examples in the GoDocs.

### Commandline

#### `-c` or `--config` (file)

Load a YAML config file. Example files can be found in the gollum directory.

#### `-n` or `--numcpu` (number)

Number of cpu cores to use.

#### `-p` or `--pidfile` (file)

Generate a pid file at a given path.

#### `-v` or `--version`

Show version and exit.

#### `-t` or `--throughput`

Write regular statistics about message / sec throughput to stdout.

#### `-cp` or `--cpuprofile` (file)

Write go CPU profiler results to a given file.

#### `-mp` or `--memprofile` (file)

Write go memory profiler results to a given file.

### Configuration file

TODO

## Use cases

TODO

### Nginx logs to Kafka

TODO

### Business logging (by PHP) to Kafka

TODO

### Accesslog parsing for Elasticsearch

TODO

### Log aggregation by many servers to files

TODO

## License

This project is released under the terms of the [Apache 2.0 license](http://www.apache.org/licenses/LICENSE-2.0).<commit_msg>Removed nginx to kafka use case to prepare this in a branch<commit_after># Gollum

Gollum is a n:m log multiplexer that gathers messages from different sources and broadcasts them to a set of listeners.

There are a few basic terms used throughout Gollum:

* A "consumer" is a plugin that reads from an external source
* A "producer" is a plugin that writes to an external source
* A "stream" is a message channel between consumer(s) and producer(s)
* A "formatter" is a plugin that adds information to a message
* A "distributor" is a plugin that routes/filters messages on a given stream

Writing a custom plugin does not require you to change any additional code besides your new plugin file.

## Consumers (reading data)

* `Console` read from stdin.
* `File` read from a file (like tail).
* `Kafka` read from a [Kafka](http://kafka.apache.org/) topic.
* `Socket` read from a socket (gollum specfic protocol).

## Producers (writing data)

* `Console` write to stdin or stdout.
* `ElasticSearch` write to [elasticsearch](http://www.elasticsearch.org/) via http/bulk.
* `File` write to a file. Supports log rotation and compression.
* `Kafka` write to a [Kafka](http://kafka.apache.org/) topic.
* `Null` like /dev/null.
* `Facebook Scribe` send messages to a [scribe](https://github.com/facebookarchive/scribe) server.
* `Socket` send messages to a socket (gollum specfic protocol).

## Formatters (modifying data)

TODO

## Distributors (multiplexing)

* `Broadcast` send to all producers in a stream.
* `Random` send to a random roducers in a stream.
* `RoundRobin` switch the producer after each send in a round robin fashion.

## Installation

### From source

Installation from source requires the installation of the [Go toolchain](http://golang.org/).

```
$ go get .
$ go build .
$ gollum --help
```

## Usage

The simplest usage is to make a local profiler run with a predefined consiguration:

```
$ gollum -c gollum_profile.conf
```

This configuration generates random messages (via *consumer.Profiler*) and write this to a file called *log_profile.log* with file rotation after 512MB per file (via *producer.File*).

## Configuration

Configuration files are written in the YAML format and have to be loaded via command line switch.
Each plugin has a different set of configuration options which are currently described in the plugin itself, i.e. you can find examples in the GoDocs.

### Commandline

#### `-c` or `--config` (file)

Load a YAML config file. Example files can be found in the gollum directory.

#### `-n` or `--numcpu` (number)

Number of cpu cores to use.

#### `-p` or `--pidfile` (file)

Generate a pid file at a given path.

#### `-v` or `--version`

Show version and exit.

#### `-t` or `--throughput`

Write regular statistics about message / sec throughput to stdout.

#### `-cp` or `--cpuprofile` (file)

Write go CPU profiler results to a given file.

#### `-mp` or `--memprofile` (file)

Write go memory profiler results to a given file.

### Configuration file

TODO

## Use cases

TODO

### Business logging (by PHP) to Kafka

TODO

### Accesslog parsing for Elasticsearch

TODO

### Log aggregation by many servers to files

TODO

## License

This project is released under the terms of the [Apache 2.0 license](http://www.apache.org/licenses/LICENSE-2.0).
{'fraction_non_alphanumeric': 0.1045682197602064, 'fraction_numerical': 0.002124753376840188, 'mean_word_length': 4.015220700152207, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '581516', 'n_tokens_mistral': 2006, 'n_tokens_neox': 1949, 'n_words': 909}