<commit_before># Yolo [![Build Status](https://travis-ci.org/ustream/yolo.png?branch=master)](https://travis-ci.org/ustream/yolo)

**You only log once but you can easily parse your data as many ways as you want.**

A general log tailer and parser tool written in Java, inspired by [Parsible](https://github.com/Yipit/parsible) and [Logster](https://github.com/etsy/logster). It is easily extensible with new parsers and processors. The main purpose was to build a simple tool to get metrics values and errors from log files and report to [StatsD](https://github.com/etsy/statsd/) (and [Graphite](http://graphite.wikidot.com/)), but everyone can write custom modules in minutes.

## Main concepts

* **Flexible configuration**: the configuration is in JSON format for easier management. You can define your parsers and processors and bind them as you want.
* **Real time**: the tool tails the log file realtime
* **Whole file reading**: with a cli parameter you can read your logfile from the beginning
* **Handle dynamic filenames**: you can use wildcards in filename (but only the first match will be used)
* **Logrotate friendly**: works easily with logrotate or other log rotating tools
* **Easily debuggable**: debug mode writes verbose logs, and you can use built-in parsers and processors for debugging purposes

## Notice

Because we tail the log file and the application can stop anytime therefore it is not guaranteed that the tool will parse all the lines. In the near future we plan to implement a secure reader which stores the read offset and handles even log rotate events.
<commit_msg>Add GitHub page link<commit_after># Yolo [![Build Status](https://travis-ci.org/ustream/yolo.png?branch=master)](https://travis-ci.org/ustream/yolo)

**You only log once but you can easily parse your data as many ways as you want.**

A general log tailer and parser tool written in Java, inspired by [Parsible](https://github.com/Yipit/parsible) and [Logster](https://github.com/etsy/logster). It is easily extensible with new parsers and processors. The main purpose was to build a simple tool to get metrics values and errors from log files and report to [StatsD](https://github.com/etsy/statsd/) (and [Graphite](http://graphite.wikidot.com/)), but everyone can write custom modules in minutes.

[GitHub Page](http://ustream.github.io/yolo/)

## Main concepts

* **Flexible configuration**: the configuration is in JSON format for easier management. You can define your parsers and processors and bind them as you want.
* **Real time**: the tool tails the log file realtime
* **Whole file reading**: with a cli parameter you can read your logfile from the beginning
* **Handle dynamic filenames**: you can use wildcards in filename (but only the first match will be used)
* **Logrotate friendly**: works easily with logrotate or other log rotating tools
* **Easily debuggable**: debug mode writes verbose logs, and you can use built-in parsers and processors for debugging purposes

## Notice

Because we tail the log file and the application can stop anytime therefore it is not guaranteed that the tool will parse all the lines. In the near future we plan to implement a secure reader which stores the read offset and handles even log rotate events.

{'fraction_non_alphanumeric': 0.08389674247080517, 'fraction_numerical': 0.0, 'mean_word_length': 4.720562390158173, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3968407', 'n_tokens_mistral': 855, 'n_tokens_neox': 828, 'n_words': 447}