<jupyter_start><jupyter_text>Word Count ExampleSpark (as Hadoop) is a massively parallel system for counting words. Although DataFrames are not the perfect data structure for implementing a word count, it is still possible.<jupyter_code>from pyspark.sql.functions import *
# Load text "Alice in wonderland"
text = spark.read.text("s3://dimajix-training/data/alice")
# Inspect Schema of "text" DataFrame
## YOUR CODE HERE
# Print first 10 entries of "text" DataFrame
## YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Extracting wordsNow we a DataFrame with a single column. Each entry contains a line of the original text file. We need to extract the individual words in three steps:1. Split each line into words using the `split` function. This will result in a DataFrame with a single column, which contains a list of words2. Convert each list of words into individual records using the `explode` function3. Remove empty words using an appropriate `filter` expression<jupyter_code># 1. Using the split function, split each record into a list of words
word_lists = text.select(split(text.value, ' ').alias("word_list"))
# 2. Using the explode function, convert each list into individual records
words = # YOUR CODE HERE
# 3. Remove empty words
non_empty_words = # YOUR CODE HERE
# Show first 10 entries
# YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Counting WordsNow that we have a DataFrame containing an individual word per record, we can count word frequencies using grouping and aggregation.1. Group by word2. Count the size of each group3. Sort by frequency (descinding)<jupyter_code># 1. & 2. Group by "word" column and count the size of each group
result = # YOUR CODE HERE

# 3. Sort words by frequency (descending)
sorted_result = # YOUR CODE HERE

# Print first 10 entries (most frequent words)
# YOUR CODE HERE<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06823027718550106, 'fraction_numerical': 0.010127931769722815, 'mean_word_length': 5.1743421052631575, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '5747', 'license': 'MIT', 'n_tokens_mistral': 542, 'n_tokens_neox': 505, 'n_words': 247, 'path': 'pyspark-ml-crashcourse/notebooks/02 - PySpark WordCount Exercise.ipynb', 'repo_name': 'Code360In/spark-code-examples', 'size': '1876'}