<commit_before>## onyx-elasticsearch

[Onyx](https://github.com/onyx-platform/onyx) plugin providing query and write for batch processing an ElasticSearch 1.x cluster.  For more details on ElasticSearch please read the [official documentation](https://www.elastic.co/guide/index.html).

**Note on the Version Number Format**:  The first three numbers in the version correspond to the latest Onyx platform release.  The final digit increments when there are changes to the plugin within an Onyx release.

### Installation

[![Clojars Project](http://clojars.org/com.liaison/onyx-elasticsearch/latest-version.svg)](http://clojars.org/com.liaison/onyx-elasticsearch)

In your project file:

```clojure
[com.liaison/onyx-elasticsearch "0.8.2.0"]
```

In your peer boot-up namespace:

```clojure
(:require [onyx.plugin.elasticsearch])
```

### Functions

#### read-messages

Reads documents from an ElasticSearch cluster with a specified query and submits them to the Onyx workflow for processing.

**Catalog entry**:

```clojure
{:onyx/name :read-messages
 :onyx/plugin :onyx.plugin.elasticsearch/read-messages
 :onyx/type :input
 :onyx/medium :elasticsearch
 :elasticsearch/host "127.0.0.1"
 :elasticsearch/port 9200
 :elasticsearch/cluster-name "my-cluster-name"
 :elasticsearch/client-type :http
 :elasticsearch/http-ops {:basic-auth ["user" "pass"]}
 :elasticsearch/index "my-index-name"
 :elasticsearch/mapping "my-mapping-name"
 :elasticsearch/query {:term {:foo "bar"}}
 :elasticsearch/sort {:foo "desc"}
 :elasticsearch/restart-on-fail false
 :onyx/batch-size batch-size
 :onyx/max-peers 1
 :onyx/doc "Read documents from an ElasticSearch Query"}
```

**Lifecycle entry**:

```clojure
{:lifecycle/task :read-messages
    :lifecycle/calls :onyx.plugin.elasticsearch/read-messages-calls}
```

**Attributes**

| key                            | type      | default     | description
|--------------------------------|-----------|-------------|-------------
|`:elasticsearch/host`           | `string`  |             | ElasticSearch Host.  Required.
|`:elasticsearch/port`           | `number`  |             | ElasticSearch Port.  Required.
|`:elasticsearch/cluster-name`   | `string`  |             | ElasticSearch Cluster Name.  Required for native connections.
|`:elasticsearch/client-type`    | `keyword` | `:http`     | Type of client to create.  Should be either `:http` or `:native`.<commit_msg>formatting fix<commit_after>## onyx-elasticsearch

[Onyx](https://github.com/onyx-platform/onyx) plugin providing query and write for batch processing an ElasticSearch 1.x cluster.  For more details on ElasticSearch please read the [official documentation](https://www.elastic.co/guide/index.html).

**Note on the Version Number Format**:  The first three numbers in the version correspond to the latest Onyx platform release.  The final digit increments when there are changes to the plugin within an Onyx release.

### Installation

[![Clojars Project](http://clojars.org/com.liaison/onyx-elasticsearch/latest-version.svg)](http://clojars.org/com.liaison/onyx-elasticsearch)

In your project file:

```clojure
[com.liaison/onyx-elasticsearch "0.8.2.0"]
```

In your peer boot-up namespace:

```clojure
(:require [onyx.plugin.elasticsearch])
```

### Functions

#### read-messages

Reads documents from an ElasticSearch cluster with a specified query and submits them to the Onyx workflow for processing.

**Catalog entry**:

```clojure
{:onyx/name :read-messages
 :onyx/plugin :onyx.plugin.elasticsearch/read-messages
 :onyx/type :input
 :onyx/medium :elasticsearch
 :elasticsearch/host "127.0.0.1"
 :elasticsearch/port 9200
 :elasticsearch/cluster-name "my-cluster-name"
 :elasticsearch/client-type :http
 :elasticsearch/http-ops {:basic-auth ["user" "<PASSWORD>"]}
 :elasticsearch/index "my-index-name"
 :elasticsearch/mapping "my-mapping-name"
 :elasticsearch/query {:term {:foo "bar"}}
 :elasticsearch/sort {:foo "desc"}
 :elasticsearch/restart-on-fail false
 :onyx/batch-size batch-size
 :onyx/max-peers 1
 :onyx/doc "Read documents from an ElasticSearch Query"}
```

**Lifecycle entry**:

```clojure
{:lifecycle/task :read-messages
 :lifecycle/calls :onyx.plugin.elasticsearch/read-messages-calls}
```

**Attributes**

| key                            | type      | default     | description
|--------------------------------|-----------|-------------|-------------
|`:elasticsearch/host`           | `string`  |             | ElasticSearch Host.  Required.
|`:elasticsearch/port`           | `number`  |             | ElasticSearch Port.  Required.
|`:elasticsearch/cluster-name`   | `string`  |             | ElasticSearch Cluster Name.  Required for native connections.
|`:elasticsearch/client-type`    | `keyword` | `:http`     | Type of client to create.  Should be either `:http` or `:native`.
{'fraction_non_alphanumeric': 0.15966735966735968, 'fraction_numerical': 0.006652806652806653, 'mean_word_length': 3.458758109360519, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5945402', 'n_tokens_mistral': 1456, 'n_tokens_neox': 1423, 'n_words': 382}