<jupyter_start><jupyter_text>kafkaSendDataPyThis notebook sends data to Kafka on the topic 'test'. A message that gives the current time is sent every second  Add dependencies<jupyter_code>import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--conf spark.ui.port=4041 --packages org.apache.kafka:kafka_2.11:0.10.0.0,org.apache.kafka:kafka-clients:0.10.0.0  pyspark-shell'<jupyter_output><empty_output><jupyter_text>Load modules and start SparkContextNote that SparkContext must be started to effectively load the package dependencies. One core is used.<jupyter_code>from pyspark import SparkContext
sc = SparkContext("local[1]", "KafkaSendStream") 
from kafka import KafkaProducer
import time<jupyter_output><empty_output><jupyter_text>Start Kafka producerOne message giving current time is sent every second to the topic test<jupyter_code>producer = KafkaProducer(bootstrap_servers='localhost:9092')
while True:
    message=time.strftime("%Y-%m-%d %H:%M:%S")
    producer.send('test', message)
    time.sleep(1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10328185328185328, 'fraction_numerical': 0.0222007722007722, 'mean_word_length': 7.038759689922481, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '137119', 'license': 'MIT', 'n_tokens_mistral': 351, 'n_tokens_neox': 340, 'n_words': 83, 'path': 'spark-streaming/kafkaSendDataPy.ipynb', 'repo_name': 'superkley/spark-playground', 'size': '1036'}