<commit_before>---
layout: post
comments: true
title: Interfacing Jython with Kafka 0.8.x
---
With the release of Kafka 0.9.0, the consumer API was redesigned to remove the dependency between consumer and Zookeeper. Prior to the 0.9.0 release, Kafka consumer was depenedent on Zookeeper for storing its offsets and the complex rebalancing logic was built right into the "high level" consumer. This was causing a couple of issues which are discussed [here](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design). Due to the issues invloving complex rebalance algoritihm within the consumer, some Kafka clients had no support for "coordinated consumption". Essentially, it means that without "coordinated consumption", you cannot have N consumers within the same consumer group consuming on the same topic which defeats one of the purpose of Kafka; to balance load across multiple consumers. I saw this problem while using `kafka-python 0.9.2` client library with `Kafka 0.8.x`. While I was trying to setup multiple consumers within the same consumer group, I noticed that every message to that topic would be reproduced on each and every consumer within that same group, defeating the purpose of Kafka being used as a traditional queing system with a pool of consumers. The issue has been discussed on [Github](https://github.com/dpkp/kafka-python/issues/199#issuecomment-53052761). 

One of the solution that I managed to figure out without moving away from Python was to use Jython for writing consumers for Kafka 0.8.x.

In this post, I would discuss about what you can do with Jython to resolve the Kafka 0.8.x issue and why it actually works. In a more traditional sense, this post would also work as a tutorial for interfacing Jython with Java.

## Setup

You need to have the following installed on your system. The tutorial assumes that you are familiar with these tools.

* Java SDK
* Python
* `virtualenv` and `virtualenvwrapper`
* Kafka 0.9.x or Kafka 0.8.x (assuming you know how to setup and run Kafka)

I have setup a bare-bones repositiory for working with this tutorial [here](https://github.com/mrafayaleem/kafka-jython/). To keep things at a minimum, `Kafka 0.9.1` binaries and `Jython 2.7.0` installer  are included within this repo. This means that you can directly run Kafa and Zookeeper after cloning this.

#### Setting up a virtualenv for Jython
Jython is fully compatible with `virtualenv` and tools such as `pip` and setting a virtualenv with Jython as the interpretter is pretty straightforward.

Jython can be installed using the GUI or console. For GUI, execute the jar and follow the steps. For installing it via console, you can use the following command to start with:

```shell<commit_msg>development: adding repo link on top<commit_after>---
layout: post
comments: true
title: Interfacing Jython with Kafka 0.8.x
---
Note: A complementing repository for this post can be found at: [https://github.com/mrafayaleem/kafka-jython](https://github.com/mrafayaleem/kafka-jython)

With the release of Kafka 0.9.0, the consumer API was redesigned to remove the dependency between consumer and Zookeeper. Prior to the 0.9.0 release, Kafka consumer was depenedent on Zookeeper for storing its offsets and the complex rebalancing logic was built right into the "high level" consumer. This was causing a couple of issues which are discussed [here](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design). Due to the issues invloving complex rebalance algoritihm within the consumer, some Kafka clients had no support for "coordinated consumption". Essentially, it means that without "coordinated consumption", you cannot have N consumers within the same consumer group consuming on the same topic which defeats one of the purpose of Kafka; to balance load across multiple consumers. I saw this problem while using `kafka-python 0.9.2` client library with `Kafka 0.8.x`. While I was trying to setup multiple consumers within the same consumer group, I noticed that every message to that topic would be reproduced on each and every consumer within that same group, defeating the purpose of Kafka being used as a traditional queing system with a pool of consumers. The issue has been discussed on [Github](https://github.com/dpkp/kafka-python/issues/199#issuecomment-53052761). 

One of the solution that I managed to figure out without moving away from Python was to use Jython for writing consumers for Kafka 0.8.x.

In this post, I would discuss about what you can do with Jython to resolve the Kafka 0.8.x issue and why it actually works. In a more traditional sense, this post would also work as a tutorial for interfacing Jython with Java.

## Setup

You need to have the following installed on your system. The tutorial assumes that you are familiar with these tools.

* Java SDK
* Python
* `virtualenv` and `virtualenvwrapper`
* Kafka 0.9.x or Kafka 0.8.x (assuming you know how to setup and run Kafka)

I have setup a bare-bones repositiory for working with this tutorial [here](https://github.com/mrafayaleem/kafka-jython/). To keep things at a minimum, `Kafka 0.9.1` binaries and `Jython 2.7.0` installer  are included within this repo. This means that you can directly run Kafa and Zookeeper after cloning this.

#### Setting up a virtualenv for Jython
Jython is fully compatible with `virtualenv` and tools such as `pip` and setting a virtualenv with Jython as the interpretter is pretty straightforward.

Jython can be installed using the GUI or console. For GUI, execute the jar and follow the steps. For installing it via console, you can use the following command to start with:

```shell
{'fraction_non_alphanumeric': 0.056325671170380766, 'fraction_numerical': 0.013335672925074574, 'mean_word_length': 4.55014605647517, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3948648', 'n_tokens_mistral': 1563, 'n_tokens_neox': 1463, 'n_words': 834}