<commit_before># A very simple Spark Steaming Example to join two streams.

## How to use
  Produce messages to kafka.

    <brokerList> is a list of kafka's blocks.

    <topics> is a kafka topics to consume from

    <keyPrefix> the prefix of key. If it is "stdin", you can input the message yoursel. format: <key,value>

    <keyNumber> the number of keys.

    <startValue> the start value of value

    <ValueNumber> the number of value.

    <interval> the interval between two messages

  Example:

    spark-submit --class com.yhd.ycache.magic.KafkaWordCountProducer   --master local \

    ./SSExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar "lujian:9092" test a 26 -100 100 100

  Consumes messages from one or more topics in Kafka and does stream joining.

    <zkQuorum> is a list of one or more zookeeper servers that make quorum

    <topics> is a list of one or more kafka topics to consume from

    <bathInterval> the interval of bath in spark stream. see spark stream docs.

    <windows> is a range of time, and we process the messages in kafka topics., see spark stream docs.

    <slide> how many 'interval' the windows move, see spark stream docs.

  Example:

     spark-submit --class com.yhd.ycache.magic.SSExample  --master local[4] ./SSExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar   \

     "jason4:2181/kafka-jason" test dest 60 4 2 2

## Use example
  ** Assuming that you has two topic 'test' and 'dest' on kafka. One of the block id is 'lujian:9092' **

  ** and zookeeper path of kafka is 'jason4:2181/kafka-jason' **

 - Produce messages of key-value to the topic 'test'
``
spark-submit --class com.yhd.ycache.magic.KafkaWordPairProducer --master local ./SSExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar "lujian:9092" test aaa 26  -100 100 100
``
 - Produce messages of key-value to the topic 'dest'
``
spark-submit --class com.yhd.ycache.magic.KafkaWordPairProducer --master local ./SSExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar "lujian:9092" dest aaa 26  100 100 100
``
 - join above two streams in Spark stream.
``
spark-submit --class com.yhd.ycache.magic.SSExample  --master local[4] ./SSExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar    "jason4:2181/kafka-jason" test dest 60 4 2 2
``<commit_msg>read me modify more readable<commit_after># Several very simple Spark Examples.

## How to use

### Produce messages to kafka.

    <brokerList> is a list of kafka's blocks.

    <topics> is a kafka topics to consume from

    <keyPrefix> the prefix of key. If it is "stdin", you can input the message yoursel. format: <key,value>

    <keyNumber> the number of keys.

    <startValue> the start value of value

    <ValueNumber> the number of value.

    <interval> the interval between two messages

  Example:

    spark-submit --class com.yhd.ycache.magic.KafkaWordCountProducer   --master local \

    ./SSExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar "lujian:9092" test a 26 -100 100 100

### Consumes messages from one or more topics in Kafka and does stream joining.

    <zkQuorum> is a list of one or more zookeeper servers that make quorum

    <topics> is a list of one or more kafka topics to consume from

    <bathInterval> the interval of bath in spark stream. see spark stream docs.

    <windows> is a range of time, and we process the messages in kafka topics., see spark stream docs.

    <slide> how many 'interval' the windows move, see spark stream docs.

  Example:

     spark-submit --class com.yhd.ycache.magic.SSExample  --master local[4] ./SSExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar   \

     "jason4:2181/kafka-jason" test dest 60 4 2 2

### Used commands of above two examples
  ** Assuming that you has two topic 'test' and 'dest' on kafka. One of the block id is 'lujian:9092' **

  ** and zookeeper path of kafka is 'jason4:2181/kafka-jason' **

 - Produce messages of key-value to the topic 'test'
``
spark-submit --class com.yhd.ycache.magic.KafkaWordPairProducer --master local ./SSExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar "lujian:9092" test aaa 26  -100 100 100
``
 - Produce messages of key-value to the topic 'dest'
``
spark-submit --class com.yhd.ycache.magic.KafkaWordPairProducer --master local ./SSExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar "lujian:9092" dest aaa 26  100 100 100
``
 - join above two streams in Spark stream.
``
spark-submit --class com.yhd.ycache.magic.SSExample  --master local[4] ./SSExample-0.0.1-SNAPSHOT-jar-with-dependencies.jar    "jason4:2181/kafka-jason" test dest 60 4 2 2
``


{'fraction_non_alphanumeric': 0.10103913331859385, 'fraction_numerical': 0.04023877957108114, 'mean_word_length': 3.2881516587677724, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4273639', 'n_tokens_mistral': 1651, 'n_tokens_neox': 1470, 'n_words': 541}