<commit_before>Kafka River Plugin for ElasticSearch
=========

The Kafka River plugin allows you to read messages from Kafka and index bulked messages into elasticsearch.
The bulk size (the number of messages to be indexed in one request) and concurrent request number is configurable.
The Kafka River also supports consuming messages from multiple Kafka brokers and multiple partitions. 

The plugin uses the latest Kafka and Elasticsearch version.
 * Kafka version 0.8.1.1
 * Elasticsearch version 1.3.2

The plugin is periodically updated, if there are newer versions of any dependencies.

Setup
==========

1. Install Kafka if you are working on local environment (See [Apacke Kafka Quick Start Guide](http://kafka.apache.org/07/quickstart.html)  for instructions on how to Download and Build.)

2. Install the plugin 

```sh
cd $ELASTICSEARCH_HOME
.bin/plugin -install <plugin-name> -url https://github.com/mariamhakobyan/elasticsearch-river-kafka/releases/download/v1.1.0/elasticsearch-river-kafka-1.1.0-plugin.zip
```
*Example:*
```sh
cd $ELASTICSEARCH_HOME
.bin/plugin -install kafka-river -url https://github.com/mariamhakobyan/elasticsearch-river-kafka/releases/download/v1.1.0/elasticsearch-river-kafka-1.1.0-plugin.zip<commit_msg>Update README (latest EL version 1.4)<commit_after>Kafka River Plugin for ElasticSearch
=========

The Kafka River plugin allows you to read messages from Kafka and index bulked messages into elasticsearch.
The bulk size (the number of messages to be indexed in one request) and concurrent request number is configurable.
The Kafka River also supports consuming messages from multiple Kafka brokers and multiple partitions. 

The plugin uses the latest Kafka and Elasticsearch version.
 * Kafka version 0.8.1.1
 * Elasticsearch version 1.4.0

The plugin is periodically updated, if there are newer versions of any dependencies.

Setup
==========

1. Install Kafka if you are working on local environment (See [Apacke Kafka Quick Start Guide](http://kafka.apache.org/07/quickstart.html)  for instructions on how to Download and Build.)

2. Install the plugin 

```sh
cd $ELASTICSEARCH_HOME
.bin/plugin -install <plugin-name> -url https://github.com/mariamhakobyan/elasticsearch-river-kafka/releases/download/v1.1.0/elasticsearch-river-kafka-1.1.0-plugin.zip
```
*Example:*
```sh
cd $ELASTICSEARCH_HOME
.bin/plugin -install kafka-river -url https://github.com/mariamhakobyan/elasticsearch-river-kafka/releases/download/v1.1.0/elasticsearch-river-kafka-1.1.0-plugin.zip
{'fraction_non_alphanumeric': 0.09848484848484848, 'fraction_numerical': 0.019138755980861243, 'mean_word_length': 4.741418764302059, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6551213', 'n_tokens_mistral': 768, 'n_tokens_neox': 735, 'n_words': 262}