<commit_before>## onyx-kafka

Onyx plugin providing read and write facilities for Kafka.

#### Installation

In your project file:

```clojure
[com.mdrogalis/onyx-kafka "0.3.1"]
```

In your peer boot-up namespace:

```clojure
(:require [onyx.plugin.kafka])
```

#### Catalog entries

##### read-messages

```clojure
{:onyx/name :read-messages
 :onyx/ident :kafka/read-messages
 :onyx/type :input
 :onyx/medium :kafka
 :onyx/consumption :sequential
 :kafka/topic topic-name
 :kafka/zookeeper "127.0.0.1:2181
 :kafka/group-id "onyx-consumer"
 :kafka/offset-reset "smallest"
 :onyx/batch-size batch-size
 :onyx/doc "Reads messages from a Kafka topic"}
```

##### write-messages

```clojure
{:onyx/name :write-messages
 :onyx/ident :kafka/write-messages
 :onyx/type :output
 :onyx/medium :kafka
 :onyx/consumption :concurrent
 :kafka/topic topic-name
 :kafka/brokers "127.0.0.1:9092"
 :kafka/serializer-class "kafka.serializer.DefaultEncoder"
 :kafka/partitioner-class "kafka.producer.DefaultPartitioner"
 :onyx/batch-size batch-size
 :onyx/doc "Reads messages from a Kafka topic"}
```

#### Attributes

|key                           | type      | description
|------------------------------|-----------|------------
|`:datomic/uri`                | `string`  | The URI of the datomic database to connect to
|`:datomic/t`                  | `integer` | The t-value of the database to read from
|`:datomic/partition`          | `keyword` | The partition of the database to read out of
|`:datomic/datoms-per-segment` | `integer` | The number of datoms to compress into a single segment

#### Contributing

Pull requests into the master branch are welcomed.

#### License

Copyright © 2014 <NAME>

Distributed under the Eclipse Public License, the same as Clojure.<commit_msg>Update README.<commit_after>## onyx-kafka

Onyx plugin providing read and write facilities for Kafka.

#### Installation

In your project file:

```clojure
[com.mdrogalis/onyx-kafka "0.3.1"]
```

In your peer boot-up namespace:

```clojure
(:require [onyx.plugin.kafka])
```

#### Catalog entries

##### read-messages

```clojure
{:onyx/name :read-messages
 :onyx/ident :kafka/read-messages
 :onyx/type :input
 :onyx/medium :kafka
 :onyx/consumption :sequential
 :kafka/topic topic-name
 :kafka/zookeeper "127.0.0.1:2181
 :kafka/group-id "onyx-consumer"
 :kafka/offset-reset "smallest"
 :onyx/batch-size batch-size
 :onyx/doc "Reads messages from a Kafka topic"}
```

##### write-messages

```clojure
{:onyx/name :write-messages
 :onyx/ident :kafka/write-messages
 :onyx/type :output
 :onyx/medium :kafka
 :onyx/consumption :concurrent
 :kafka/topic topic-name
 :kafka/brokers "127.0.0.1:9092"
 :kafka/serializer-class "kafka.serializer.DefaultEncoder"
 :kafka/partitioner-class "kafka.producer.DefaultPartitioner"
 :onyx/batch-size batch-size
 :onyx/doc "Reads messages from a Kafka topic"}
```

#### Attributes

|key                           | type      | description
|------------------------------|-----------|------------
|`:kafka/topic`                | `string`  | The topic name to connect to
|`:kafka/zookeeper`            | `string`  | The ZooKeeper connection string
|`:kafka/group-id`             | `string`  | The consumer identity to store in ZooKeeper
|`:kafka/offset-reset`         | `string`  | Offset to seek to when not found - "smallest" or "largest"
|`:kafka/brokers`              | `string`  | A Kafka brokers connection string
|`:kafka/serializer-class`     | `string`  | The Kafka serialization class to use
|`:kafka/partitioner-class`    | `string`  | The Kafka partitioning class to use

#### Contributing

Pull requests into the master branch are welcomed.

#### License

Copyright © 2014 <NAME>

Distributed under the Eclipse Public License, the same as Clojure.
{'fraction_non_alphanumeric': 0.1621477937267411, 'fraction_numerical': 0.014354066985645933, 'mean_word_length': 3.2376126126126126, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3646774', 'n_tokens_mistral': 1311, 'n_tokens_neox': 1226, 'n_words': 332}