<commit_before>Status
======

This is a *spike* to learn Storm and how to integrate it with Jython and other tech. For now, that means no testing
and minimal docs.


Setup
=====

This will be replaced by a much better setup. But for now:

You need Storm installed with its bin directory on `$PATH`.

Checkout Jython trunk and build it. It includes a necessary fix for using Jython with Clojure types as well as new custom proxy maker support for better Java integration.


~~~~
$ hg clone http://hg.python.org/jython jython27
$ cd jython27 
$ ant                                                 # build development version (fastest way)
$ export PATH=$(pwd)/dist/bin:$PATH                   # add jython to your path
~~~~

This package also includes a very preliminary and ultra simplistic implementation of clamp for Java <=> Python integration, similar to what is discussed here: http://darjus.blogspot.com/2013/01/customizing-jython-proxymaker.html. One key difference: this version is written in pure Jython (that is, only Python, importing standard Java libraries, but otherwise not in Java).

Two more steps:

1. Build an "uber" jar file for Storm; this contains all necessary jars. Running `storm classpath` computes the necessary classpath for Storm dependencies:

~~~~
CLASSPATH="$(storm classpath)" proxymaker-jython27 gen-storm-jar.py -o uber.jar -i romper -i clamp --proxy=romper.topology
~~~~

2. Run the topology. Choose standalone mode with `--local`. In both cases, the cluster/standalone submitting code ("main") is in `__run__.py`:

~~~~
CLASSPATH="$(storm classpath):$(pwd)/uber.jar" java org.python.util.JarRunner --local
~~~~

or on the storm cluster by submitting the uber jar:

~~~~
$ storm jar uber.jar org.python.util.JarRunner<commit_msg>Remove proxymaker bits<commit_after>Status
======

This is a *spike* to learn Storm and how to integrate it with Jython and other tech. For now, that means no testing
and minimal docs.


Setup
=====

This will be replaced by a much better setup. But for now:

You need Storm installed with its bin directory on `$PATH`.

Checkout Jython trunk and build it. It includes a necessary fix for using Jython with Clojure types as well as new custom proxy maker support for better Java integration.


~~~~
$ hg clone http://hg.python.org/jython jython27
$ cd jython27 
$ ant                                                 # build development version (fastest way)
$ export PATH=$(pwd)/dist/bin:$PATH                   # add jython to your path; I sometimes alias to jython27 to be clear
~~~~

This package also includes a very preliminary and ultra simplistic implementation of clamp for Java <=> Python integration, similar to what is discussed here: http://darjus.blogspot.com/2013/01/customizing-jython-proxymaker.html. One key difference: this version is written in pure Jython (that is, only Python, importing standard Java libraries, but otherwise not in Java).

Two more steps:

1. Build an "uber" jar file for Storm; this contains all necessary jars. Running `storm classpath` computes the necessary classpath for Storm dependencies:

~~~~
CLASSPATH="$(storm classpath)" jython gen-storm-jar.py -o uber.jar -i romper -i clamp --proxy=romper.topology
~~~~

2. Run the topology. Choose standalone mode with `--local`. In both cases, the cluster/standalone submitting code ("main") is in `__run__.py`:

~~~~
CLASSPATH="$(storm classpath):$(pwd)/uber.jar" java org.python.util.JarRunner --local
~~~~

or on the storm cluster by submitting the uber jar:

~~~~
$ storm jar uber.jar org.python.util.JarRunner
{'fraction_non_alphanumeric': 0.0941968040370059, 'fraction_numerical': 0.00784973366975049, 'mean_word_length': 3.6824146981627295, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4371922', 'n_tokens_mistral': 1026, 'n_tokens_neox': 967, 'n_words': 451}