<jupyter_start><jupyter_text>PySpark Demo Notebook Demo1. Setup Spark2. Load Kaggle Data3. Transform Data with Spark SQL_Prepared by: [<NAME>](https://twitter.com/GaryStafford)   Associated article: https://wp.me/p1RD28-61V_  Setup SparkSetup Spark SparkSession<jupyter_code>from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
spark = SparkSession \
    .builder \
    .getOrCreate()<jupyter_output><empty_output><jupyter_text>Load Kaggle DataLoad the Kaggle dataset from the CSV file, containing ~21K records, into a DataFrame<jupyter_code>bakery_schema = StructType([
    StructField('date', StringType(), True),
    StructField('time', StringType(), True),
    StructField('transaction', IntegerType(), True),
    StructField('item', StringType(), True)
])

df_bakery1 = spark.read \
    .format("csv") \
    .option("header", "true") \
    .load("s3://gstafford-aws-emr-notebooks/files/BreadBasket_DMS.csv", schema=bakery_schema)

df_bakery1.show(10)
df_bakery1.count()<jupyter_output><empty_output><jupyter_text>Transform Data with Spark SQLTransform the DataFrame's bakery data using Spark SQL<jupyter_code>df_bakery1.createOrReplaceTempView("bakery_table_tmp1")

df_bakery2 = spark.sql("SELECT date, transaction, item " +
                       "FROM bakery_table_tmp1 " +
                       "WHERE item NOT LIKE 'NONE'" +
                       "ORDER BY transaction")
df_bakery2.show(5)
df_bakery2.count()
df_bakery2.createOrReplaceTempView("bakery_table_tmp2")

df_bakery3 = spark.sql("SELECT date, count(*) as count " +
                       "FROM bakery_table_tmp2 " +
                       "WHERE date >= '2017-01-01' " +
                       "GROUP BY date " +
                       "ORDER BY date")
df_bakery3.show(5)
df_bakery3.count()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11542610571736785, 'fraction_numerical': 0.020496224379719527, 'mean_word_length': 3.672544080604534, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 15, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '223168', 'license': 'MIT', 'n_tokens_mistral': 652, 'n_tokens_neox': 613, 'n_words': 122, 'path': 'work/05_pyspark_demo_notebook_emr.ipynb', 'repo_name': 'Richaro10/pyspark-setup-demo', 'size': '1864'}