<commit_before># learning-kafka
Just an attempt to learn kafka and its integration with Spark

#### How to run spark kafka integration directly from command line (or IDE, pycharm in my example)

1. Download `http://search.maven.org/remotecontent?filepath=org/apache/spark/spark-streaming-kafka-assembly_2.10/1.3.1/spark-streaming-kafka-assembly_2.10-1.3.1.jar`
2. Copy it to `SPARK_HOME/lib`
3. edit `SPARK_HOME/bin/compute_classpath.sh`
4. add the following line right before the last line (the print statement in the file)
  * `appendToClasspath "$FWDIR"/lib/spark-streaming-kafka-assembly_2.10-1.3.1.jar`
5. test that kafka assembly will get picked up by spark just run 
  * SPARK_HOME/bin/compute_classpath.sh 
  * it should look like (basically have the kafka assemly jar in the list)
    * `/usr/spark/conf:/usr/spark/lib/spark-assembly-1.3.1-hadoop2.6.0.jar:/usr/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/spark/lib/datanucleus-core-3.2.10.jar:/usr/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/spark/lib/spark-streaming-kafka-assembly_2.10-1.3.1.jar`
	
5. to your ~/.bashrc append the following lines
  * `export SPARK_HOME=/usr/spark`
  * save and exit

6. add pyspark to PYTHONPATH There are may ways to do it. I personally like this (http://stackoverflow.com/a/12311321) <commit_msg>minor edits to readme (formatting)<commit_after># learning-kafka
Just an attempt to learn kafka and its integration with Spark

#### How to run spark kafka integration directly from command line (or IDE, pycharm in my example)

1. Download http://search.maven.org/remotecontent?filepath=org/apache/spark/spark-streaming-kafka-assembly_2.10/1.3.1/spark-streaming-kafka-assembly_2.10-1.3.1.jar
2. Copy it to `SPARK_HOME/lib`
3. edit `SPARK_HOME/bin/compute_classpath.sh`
4. add the following line right before the last line (the print statement in the file)
  * `appendToClasspath "$FWDIR"/lib/spark-streaming-kafka-assembly_2.10-1.3.1.jar`
5. test that kafka assembly will get picked up by spark just run 
  * SPARK_HOME/bin/compute_classpath.sh 
  * it should look like (basically have the kafka assemly jar in the list)
    * `/usr/spark/conf:/usr/spark/lib/spark-assembly-1.3.1-hadoop2.6.0.jar:/usr/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/spark/lib/datanucleus-core-3.2.10.jar:/usr/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/spark/lib/spark-streaming-kafka-assembly_2.10-1.3.1.jar`
	
5. to your ~/.bashrc append the following lines
  * `export SPARK_HOME=/usr/spark`
  * save and exit

6. add pyspark to PYTHONPATH There are may ways to do it. I personally like this http://stackoverflow.com/a/12311321 
{'fraction_non_alphanumeric': 0.1244717633499808, 'fraction_numerical': 0.042258932001536686, 'mean_word_length': 3.998080614203455, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6997197', 'n_tokens_mistral': 1028, 'n_tokens_neox': 941, 'n_words': 236}