<jupyter_start><jupyter_text>Word count with PySpark Topics covered in this example* Write a file to HDFS, read the file and perform word count on the data. *** PrerequisitesNOTE : In order to execute this notebook successfully as is, please ensure the following prerequisites are completed.* The EMR cluster attached to this notebook should have the `Spark` application installed.* This notebook uses the `PySpark` kernel.***  IntroductionIn this example we write a file to hdfs, use pyspark to count the occurrence of each word in the file stored in hdfs and store the results to s3.***  Setup1. Create a S3 bucket to save your results or use an existing s3 bucket. For example: `s3://EXAMPLE-BUCKET/word-count/`***  ExampleCreate a test data frame with some sample records.We will use the `createDataFrame()` method to create and `printSchema()` method to print out the schema.<jupyter_code>wordsDF = sqlContext.createDataFrame([("emr",), ("spark",), ("example",), ("spark",), ("pyspark",), ("python",),
             ("example",), ("emr",), ("example",), ("spark",), ("pyspark",), ("python",)], ["words"])
wordsDF.show()
wordsDF.printSchema()<jupyter_output><empty_output><jupyter_text>Print out the number of unique words so that we can verify this number with the end result count.<jupyter_code>uniqueWordsCount = wordsDF.distinct().groupBy().count().head()[0]
print(uniqueWordsCount)<jupyter_output><empty_output><jupyter_text>This step only shows an example on how to write to hdfs.You can use an existing file stored in hdfs and read it as shown in the next steps.<jupyter_code>wordsDF.write.csv("hdfs:///user/hadoop/test-data.csv")<jupyter_output><empty_output><jupyter_text>Read the csv file from hdfs and store in RDD.<jupyter_code>wordsData = sc.textFile("hdfs:///user/hadoop/test-data.csv")
wordsData.count()<jupyter_output><empty_output><jupyter_text>Display the contents of the file.<jupyter_code>wordsData.collect()<jupyter_output><empty_output><jupyter_text>Count the occurance of each word and print the count of the result. This should be equal to the number of unique words we found earlier.<jupyter_code>wordsCounts = wordsData.flatMap(lambda line: line.split(" ")) \
                       .map(lambda word: (word, 1)) \
                       .reduceByKey(lambda a, b: a+b)
wordsCounts.count()<jupyter_output><empty_output><jupyter_text>Display the count for each word.<jupyter_code>wordsCounts.collect()<jupyter_output><empty_output><jupyter_text>Save the results to your s3 bucket. The results are stored in the key `word-count` and split based on paritions.<jupyter_code>wordsCounts.saveAsTextFile("s3://EXAMPLE-BUCKET/word-count") # Change this to the S3 location that you created in Setup step 1.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11328976034858387, 'fraction_numerical': 0.0039941902687000725, 'mean_word_length': 5.497641509433962, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '125266', 'license': 'MIT-0', 'n_tokens_mistral': 845, 'n_tokens_neox': 824, 'n_words': 285, 'path': 'examples/word-count-with-pyspark.ipynb', 'repo_name': 'cyn0/emr-studio-notebook-examples', 'size': '2754'}