<commit_before>#####
samsa
#####

    One morning, as <NAME> was waking up from anxious dreams, he
    discovered that in bed he had been changed into a monstrous verminous bug.

Samsa is a cluster-aware Kafka protocol client and routing library for Python.

************
Introduction
************

What is Kafka?
==============

Kafka is a distributed publish-subscribe messaging system. Cluster management
happens with ZooKeeper.

* Brokers
* Topics
* Partitions
* Messages

The clients of a Kafka cluster are generally split into two different categories:

* Producer
* Consumer

Goals of this Project
=====================

* Your application code should not be dependent on broker state or Kafka cluster
  configuration. The only service that it should know about directly is the
  ZooKeeper cluster.
* Provide a level of abstraction and functionality similar to that of the Scala
  client, but using Pythonic idioms and native data structures.

Guarantees
==========

* Every publish will be written to (at least) one node, as long as a node is up
  to accept the write.

**********************
Usage and API Overview
**********************

Creating a Cluster
==================

::

    >>> from kazoo.client import KazooClient
    >>> from samsa.cluster import Cluster
    >>> zookeeper = KazooClient()
    >>> zookeeper.connect()
    >>> cluster = Cluster(zookeeper)

Brokers
=======

To list all of the brokers by their broker ID within the cluster:

::

    >>> print cluster.brokers.keys()
    [0, 1]

To get a ``Broker`` object by ID, provide the ID as the key in a dictionary
item lookup::

    >>> print cluster.brokers[0]
    <samsa.brokers.Broker at 0x1005f4c10: id=0>

Topics
======

``Cluster`` objects also provide a dictionary-like interface for retrieving
``Topic`` objects by name. For example::

    >>> topic = cluster.topics['example-topic']
    >>> print topic
    <samsa.topics.Topic at 0x1005f4d90: name='example-topic'>

You can also see how many partitions are available to accept writes within a
topic by coercing it's ``PartitionMap`` to a list::

    >>> print list(topic.partitions)
    [<samsa.partitions.Partition at 0x1005f9b90: topic=<samsa.topics.Topic at 0x1005f4d90: name='example-topic'>, broker=<samsa.brokers.Broker at 0x1005f4c10: id=0>, number=0>]

Publishing
==========

To publish to a topic, provide a string or list of strings to be published to
a ``Topic`` instances's ``publish`` method::

    >>> topic.publish('hello world')
    >>> topic.publish(['hello', 'world'])

If a list of message is provided, all messages will be delivered to the same
partition, in order, on the same broker. Individual messages hold no such
guarantee, and are subject to end up on any number of hosts as determined by
the partitioning method, and can be distributed across a number of partitions,
potentially losing any ordering between messages.

Consuming
=========

Consumers are organized into consumer groups, which allows Kafka to provide two
semantically different methods of message consumption, based on the consumer
group configuration.

Queue
    Each consumer in a consumer group receives a message once, which
    provides for a reasonably even distribution of messages around the group
    since it is the owner of only a subset of the available partitions within
    the topic. This is similar to a put/get queue or many AMQP-like interfaces.
Topic
    Each consumer is it's own consumer group, and receives all messages
    sent to the topic since it is the owner of all partitions. This is similar
    to an evented system, or a publish/subscribe interface.

A Note about Partition Counts
-----------------------------

An important thing to note when configuring your broker is that if there are
more consumers than partitions, some consumers won't get any messages at all,
so you should configure your brokers up front to split topics into at least as
many partitions as you expect to have consumers.
<commit_msg>fix typo<commit_after>#####
samsa
#####

    One morning, as <NAME> was waking up from anxious dreams, he
    discovered that in bed he had been changed into a monstrous verminous bug.

Samsa is a cluster-aware Kafka protocol client and routing library for Python.

************
Introduction
************

What is Kafka?
==============

Kafka is a distributed publish-subscribe messaging system. Cluster management
happens with ZooKeeper.

* Brokers
* Topics
* Partitions
* Messages

The clients of a Kafka cluster are generally split into two different categories:

* Producer
* Consumer

Goals of this Project
=====================

* Your application code should not be dependent on broker state or Kafka cluster
  configuration. The only service that it should know about directly is the
  ZooKeeper cluster.
* Provide a level of abstraction and functionality similar to that of the Scala
  client, but using Pythonic idioms and native data structures.

Guarantees
==========

* Every publish will be written to (at least) one node, as long as a node is up
  to accept the write.

**********************
Usage and API Overview
**********************

Creating a Cluster
==================

::

    >>> from kazoo.client import KazooClient
    >>> from samsa.cluster import Cluster
    >>> zookeeper = KazooClient()
    >>> zookeeper.connect()
    >>> cluster = Cluster(zookeeper)

Brokers
=======

To list all of the brokers by their broker ID within the cluster:

::

    >>> print cluster.brokers.keys()
    [0, 1]

To get a ``Broker`` object by ID, provide the ID as the key in a dictionary
item lookup::

    >>> print cluster.brokers[0]
    <samsa.brokers.Broker at 0x1005f4c10: id=0>

Topics
======

``Cluster`` objects also provide a dictionary-like interface for retrieving
``Topic`` objects by name. For example::

    >>> topic = cluster.topics['example-topic']
    >>> print topic
    <samsa.topics.Topic at 0x1005f4d90: name='example-topic'>

You can also see how many partitions are available to accept writes within a
topic by coercing it's ``PartitionMap`` to a list::

    >>> print list(topic.partitions)
    [<samsa.partitions.Partition at 0x1005f9b90: topic=<samsa.topics.Topic at 0x1005f4d90: name='example-topic'>, broker=<samsa.brokers.Broker at 0x1005f4c10: id=0>, number=0>]

Publishing
==========

To publish to a topic, provide a string or list of strings to be published to
a ``Topic`` instances's ``publish`` method::

    >>> topic.publish('hello world')
    >>> topic.publish(['hello', 'world'])

If a list of messages is provided, all messages will be delivered to the same
partition, in order, on the same broker. Individual messages hold no such
guarantee, and are subject to end up on any number of hosts as determined by
the partitioning method, and can be distributed across a number of partitions,
potentially losing any ordering between messages.

Consuming
=========

Consumers are organized into consumer groups, which allows Kafka to provide two
semantically different methods of message consumption, based on the consumer
group configuration.

Queue
    Each consumer in a consumer group receives a message once, which
    provides for a reasonably even distribution of messages around the group
    since it is the owner of only a subset of the available partitions within
    the topic. This is similar to a put/get queue or many AMQP-like interfaces.
Topic
    Each consumer is it's own consumer group, and receives all messages
    sent to the topic since it is the owner of all partitions. This is similar
    to an evented system, or a publish/subscribe interface.

A Note about Partition Counts
-----------------------------

An important thing to note when configuring your broker is that if there are
more consumers than partitions, some consumers won't get any messages at all,
so you should configure your brokers up front to split topics into at least as
many partitions as you expect to have consumers.

{'fraction_non_alphanumeric': 0.10771947527749748, 'fraction_numerical': 0.01160443995963673, 'mean_word_length': 4.082692307692308, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 87, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5744698', 'n_tokens_mistral': 2175, 'n_tokens_neox': 2034, 'n_words': 1041}