<commit_before>Petrel
======

Tools for writing, submitting, debugging, and monitoring Storm topologies in pure Python.

Overview
========

While Storm has basic support for any Thrift-compatible language, that support is quite minimal:

* Storm communicates with other languages by sending JSON messages through stdin and stdout. This makes it difficult to use pdb for debugging. Also, this design may not offer the best performance.
* There is no easy way to supply configuration information when submitting a topology.
* The Storm client for Python supplied in storm-starter is very basic.
** It supports implementing spouts and bolts but not describing topologies. Thus you have to write Java code to describe your Python topology.
** It has little or no support for including Python packages (.tar.gz or .egg files) with a topology.
** It has no support for testing spouts or bolts outside of Storm.
** It has potential performance issues (e.g. building a dictionary by inserting one key at a time).
** It has no logging. If a spout or bolt raises an exception, the task will die without capturing any information about what happened or why.

Petrel addresses these issues.

Topology definition
===================

<pre>
import randomsentence
import splitsentence
import wordcount

def create(builder):
    builder.setSpout("spout", randomsentence.RandomSentenceSpout(), 1)
    builder.setBolt("split", splitsentence.SplitSentenceBolt(), 1).shuffleGrouping("spout")
    builder.setBolt("count", wordcount.WordCountBolt(), 1).fieldsGrouping("split", ["word"])
</pre>

Building and submitting topologies
==================================

Use the following command to package and submit a topology to Storm:

<pre>
petrel submit --sourcejar ../../jvmpetrel/target/storm-petrel-*-SNAPSHOT.jar --config localhost.yaml wordcount
</pre>
<commit_msg>Fixed enumeration<commit_after>Petrel
======

Tools for writing, submitting, debugging, and monitoring Storm topologies in pure Python.

Overview
========

While Storm has basic support for any Thrift-compatible language, that support is quite minimal:

* Storm communicates with other languages by sending JSON messages through stdin and stdout. This makes it difficult to use pdb for debugging. Also, this design may not offer the best performance.
* There is no easy way to supply configuration information when submitting a topology.
* The Storm client for Python supplied in storm-starter is very basic.
 * It supports implementing spouts and bolts but not describing topologies. Thus you have to write Java code to describe your Python topology.
 * It has little or no support for including Python packages (.tar.gz or .egg files) with a topology.
 * It has no support for testing spouts or bolts outside of Storm.
 * It has potential performance issues (e.g. building a dictionary by inserting one key at a time).
 *It has no logging. If a spout or bolt raises an exception, the task will die without capturing any information about what happened or why.

Petrel addresses these issues.

Topology definition
===================

<pre>
import randomsentence
import splitsentence
import wordcount

def create(builder):
    builder.setSpout("spout", randomsentence.RandomSentenceSpout(), 1)
    builder.setBolt("split", splitsentence.SplitSentenceBolt(), 1).shuffleGrouping("spout")
    builder.setBolt("count", wordcount.WordCountBolt(), 1).fieldsGrouping("split", ["word"])
</pre>

Building and submitting topologies
==================================

Use the following command to package and submit a topology to Storm:

<pre>
petrel submit --sourcejar ../../jvmpetrel/target/storm-petrel-*-SNAPSHOT.jar --config localhost.yaml wordcount
</pre>

{'fraction_non_alphanumeric': 0.10234944639481501, 'fraction_numerical': 0.0016203078584931137, 'mean_word_length': 4.935897435897436, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2153997', 'n_tokens_mistral': 961, 'n_tokens_neox': 899, 'n_words': 453}