<commit_before>flume-ng-elasticsearch-serializer-num
=====================================

An elasticsearch flume serializer with awareness for numerical value.
This is a quick and dirty hack to support numerical values stored to elasticsearch in order to analyze it with Kibana.
Please be aware that this implementation might not perform the very best and only use it with caution :)

Config
------
In order to attach this serializer to flume the following tasks need to be performed:
1. `mvn clean assembly:assembly`
2. copy the generated jar (\*-with-dependencies-\*) into the flume lib directory
3. define the following serializer: ch.creativecode.logging.ElasticsearchNumSerializer (e.g. `a1.sinks.k1.serializer = ch.creativecode.logging.ElasticsearchNumSerializer`)

<commit_msg>readme structure<commit_after>flume-ng-elasticsearch-serializer-num
=====================================

An elasticsearch flume serializer with awareness for numerical value.
This is a quick and dirty hack to support numerical values stored to elasticsearch in order to analyze it with Kibana.
Please be aware that this implementation might not perform the very best and only use it with caution :)

Config
------
In order to attach this serializer to flume the following tasks need to be performed:  
1. `mvn clean assembly:assembly`  
2. copy the generated jar (\*-with-dependencies-\*) into the flume lib directory  
3. define the following serializer: ch.creativecode.logging.ElasticsearchNumSerializer (e.g. `a1.sinks.k1.serializer = ch.creativecode.logging.ElasticsearchNumSerializer`)  


{'fraction_non_alphanumeric': 0.10991787744788377, 'fraction_numerical': 0.006317119393556538, 'mean_word_length': 5.0, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3328267', 'n_tokens_mistral': 397, 'n_tokens_neox': 397, 'n_words': 179}