<commit_before># Logfan

Logstash like implementation with GO.

![Logfan logo](docs/static/noun_307496_cc.png "Logfan")

## Install
2 ways to get logfan : download a released version or compile it from source.

### Download binary
linux, windows, osx available here : https://github.com/veino/logfan/releases

### Get source and compile
```
$ go get -u github.com/veino/logfan
```

## Usage
type `logfan help` to display usage information

	logstash flags works `-f`, `-e`, `--configtest`, ...

## Run 
```
$ logfan run https://raw.githubusercontent.com/veino/logfan/master/examples.d/simple.conf
```

copy/paste this in your console

```
127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
```

## Use configuration from another location 
### include configuration from an URL
```
$ logfan run "input{stdin{}} filter{use{url=>'https://raw.githubusercontent.com/veino/logfan/master/examples.d/use/lol/test.conf'}} output{stdout{codec=>rubydebug}}"
```

## use configuration file on local filesystem
```
$ logfan run "input{stdin{}} filter{use{path=>'apachelogs.conf'}} output{stdout{codec=>rubydebug}}"
```

See examples in examples.d/use/ folder

### TODO

- [x] parse logstash config file
- [x] support command line flags "à la logstash"
- [x] generic input support
- [x] generic filter support
- [x] generic output support
- [x] configuration condition (if else) support
- [x] dynamic %{field.key} support in config file
- [x] gracefully stop
- [x] gracefully start
- [x] name all contributors, imported packages, similar projects
- [x] use remote configuration file
- [x] include local and remote files from configuration files
- [ ] codec support
- [ ] log to file<commit_msg>update README (thinking about building a documentation website)<commit_after># Logfan

Logstash like implementation with GO.

![Logfan logo](docs/static/noun_307496_cc.png "Logfan")

## Install
2 ways to get logfan : download a released version or compile it from source.

### Download binary
linux, windows, osx available here : https://github.com/veino/logfan/releases

### Get source and compile
```
$ go get -u github.com/veino/logfan
```

## Run 
```
$ logfan run https://raw.githubusercontent.com/veino/logfan/master/examples.d/simple.conf
```

	logstash flags works as well `-f`, `-e`, `--configtest`, ...

copy/paste this in your console

```
127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
```

## Other command
type `logfan help` to display usage information

	
```
Usage:
  logfan [flags]
  logfan [command]

Available Commands:
  doc         Display documentation about plugins
  run         Run logfan
  service     Install and manage logfan service
  test        Test configurations (files, url, directories)
  version     Display version informations

Flags:
  -f, --config string       Load the Logstash config from a file a directory or a url
  -t, --configtest          Test config file or directory
      --debug               Increase verbosity to the last level (trace), more verbose.
  -e, --eval string         Use the given string as the configuration data.
  -w, --filterworkers int   number of workers (default 4)
  -l, --log string          Log to a given path. Default is to log to stdout.
      --verbose             Increase verbosity to the first level (info), less verbose.
  -V, --version             Display version info.

Use "logfan [command] --help" for more information about a command.
```

## include configuration from other configuration
### include configuration from an URL
```
$ logfan run "input{stdin{}} filter{use{url=>'https://raw.githubusercontent.com/veino/logfan/master/examples.d/use/lol/test.conf'}} output{stdout{codec=>rubydebug}}"
```

## use configuration file on local filesystem
```
$ logfan run "input{stdin{}} filter{use{path=>'apachelogs.conf'}} output{stdout{codec=>rubydebug}}"
```

See examples in examples.d/use/ folder

## TODO

- [x] parse logstash config file
- [x] support command line flags "à la logstash"
- [x] generic input support
- [x] generic filter support
- [x] generic output support
- [x] configuration condition (if else) support
- [x] dynamic %{field.key} support in config file
- [x] gracefully stop
- [x] gracefully start
- [x] name all contributors, imported packages, similar projects
- [x] use remote configuration file
- [x] include local and remote files from configuration files
- [ ] codec support
- [ ] log to file
{'fraction_non_alphanumeric': 0.1293967171178853, 'fraction_numerical': 0.024515028778512044, 'mean_word_length': 3.7298387096774195, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3882276', 'n_tokens_mistral': 1571, 'n_tokens_neox': 1451, 'n_words': 503}