<commit_before># buttered-scones [![Build Status](https://travis-ci.org/alindeman/buttered-scones.svg?branch=master)](https://travis-ci.org/alindeman/buttered-scones)

**buttered-scones** watches log files on disk and forwards them to **logstash**
via the **lumberjack** protocol. It is similar to
[**logstash-forwarder**](https://github.com/elasticsearch/logstash-forwarder).

> I'm a lumberjack and I'm OK

> I sleep all night and I work all day.

> [...]

> On Wednesdays I go shoppin'

> And have **buttered scones** for tea.

## Configuration

Like **logstash-forwarder**, **buttered-scones** is configured via a JSON file.

```json
{
  "state": "/var/lib/buttered-scones/state.db",

  "network": {
    "server":       "logstash.internal.example.com:5043",
    "certificate":  "/etc/buttered-scones/forwarder.crt",
    "key":          "/etc/buttered-scones/forwarder.key",
    "ca":           "/etc/buttered-scones/ca.crt",
    "timeout":      15
  },

  "statistics": {
    "addr": ":8088"
  },

  "files": [
    {
      "paths":  ["/var/log/messages", "/var/log/*.log"],
      "fields": {"type": "syslog"}
    }
  ]
}
```

**state** is where **buttered-scones** keeps information about how far it has
read into each file. The directory where it lives must be writable by the
user that runs the **buttered-scones** process.

The SSL certificate presented by the remote logstash server must be signed by
the specified CA, if the `"ca"` option is specified. Otherwise,
**buttered-scones** will not communicate with the remote server.

If given, **statistics/addr** specifies a socket address where an HTTP server
will listen. Statistics about what **buttered-scones** is doing will be written
in JSON format. Use these statistics to debug problems or write automated
monitoring tools. For example: `curl -si http://localhost:8088`

**files** supports glob patterns. **buttered-scones** will periodically check
for new files that match the glob pattern and tail them.

Currently, **buttered-scones** does _not_ support log files that are truncated
or renamed. This is not a use case the original developers had. However, if it
interests you, pull requests are welcomed.

## Development & Packaging

To build the static binary, `buttered-scones`:

```
script/build
```

To run the tests:

```
script/test
```

To package `buttered-scones` into a debian package:

```
GOOS=linux GOARCH=amd64 VERSION=0.0.1 script/deb
```

## Future Work

* Support multiple servers
* Support input from standard in
* Support files which are truncated in place<commit_msg>Documents a gotcha in Supervisor right now<commit_after># buttered-scones [![Build Status](https://travis-ci.org/alindeman/buttered-scones.svg?branch=master)](https://travis-ci.org/alindeman/buttered-scones)

**buttered-scones** watches log files on disk and forwards them to **logstash**
via the **lumberjack** protocol. It is similar to
[**logstash-forwarder**](https://github.com/elasticsearch/logstash-forwarder).

> I'm a lumberjack and I'm OK

> I sleep all night and I work all day.

> [...]

> On Wednesdays I go shoppin'

> And have **buttered scones** for tea.

## Configuration

Like **logstash-forwarder**, **buttered-scones** is configured via a JSON file.

```json
{
  "state": "/var/lib/buttered-scones/state.db",

  "network": {
    "server":       "logstash.internal.example.com:5043",
    "certificate":  "/etc/buttered-scones/forwarder.crt",
    "key":          "/etc/buttered-scones/forwarder.key",
    "ca":           "/etc/buttered-scones/ca.crt",
    "timeout":      15
  },

  "statistics": {
    "addr": ":8088"
  },

  "files": [
    {
      "paths":  ["/var/log/messages", "/var/log/*.log"],
      "fields": {"type": "syslog"}
    }
  ]
}
```

**state** is where **buttered-scones** keeps information about how far it has
read into each file. The directory where it lives must be writable by the
user that runs the **buttered-scones** process.

The SSL certificate presented by the remote logstash server must be signed by
the specified CA, if the `"ca"` option is specified. Otherwise,
**buttered-scones** will not communicate with the remote server.

If given, **statistics/addr** specifies a socket address where an HTTP server
will listen. Statistics about what **buttered-scones** is doing will be written
in JSON format. Use these statistics to debug problems or write automated
monitoring tools. For example: `curl -si http://localhost:8088`

**files** supports glob patterns. **buttered-scones** will periodically check
for new files that match the glob pattern and tail them.

Currently, **buttered-scones** does _not_ support log files that are truncated
or renamed. This is not a use case the original developers had. However, if it
interests you, pull requests are welcomed.

## Development & Packaging

To build the static binary, `buttered-scones`:

```
script/build
```

To run the tests:

```
script/test
```

To package `buttered-scones` into a debian package:

```
GOOS=linux GOARCH=amd64 VERSION=0.0.1 script/deb
```

## Future Work

* Support multiple servers
* Support input from standard in
* Support files which are truncated in place

### Gotchas

* `Supervisor` doesn't clean up its readers or spooler quite correctly, though this does not matter much right now because the program always exits immediately after the Supervisor is shut down. However, it'd be nicer if we could cleanly close all of the channels (and therefore goroutines) while also making sure we don't panic by sending to a closed channel in one of the readers.
{'fraction_non_alphanumeric': 0.1233742774566474, 'fraction_numerical': 0.006864161849710983, 'mean_word_length': 3.9703770197486534, 'pattern_counts': {'":': 28, '<': 3, '<?xml version=': 0, '>': 13, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5156352', 'n_tokens_mistral': 1763, 'n_tokens_neox': 1697, 'n_words': 628}