<jupyter_start><jupyter_text>PySpark Demo Plotly Notebook Demo1. Setup Spark2. Load Kaggle Data3. Transform Data with Spark SQL4. Graph Data with Plotly_Prepared by: [<NAME>](https://twitter.com/GaryStafford)   Associated article: https://wp.me/p1RD28-61V_  Setup SparkSetup Spark SparkSession<jupyter_code>! pip install pip --upgrade --quiet
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
spark = SparkSession \
    .builder \
    .appName('pyspark_demo_app') \
    .master("local[*]") \
    .getOrCreate()<jupyter_output><empty_output><jupyter_text>Load Kaggle DataLoad the Kaggle dataset from the CSV file, containing ~21K records, into a DataFrame<jupyter_code>bakery_schema = StructType([
    StructField('date', StringType(), True),
    StructField('time', StringType(), True),
    StructField('transaction', IntegerType(), True),
    StructField('item', StringType(), True)
])

df_bakery1 = spark.read \
    .format("csv") \
    .option("header", "true") \
    .load("BreadBasket_DMS.csv", schema=bakery_schema)

df_bakery1.show(10)
df_bakery1.count()<jupyter_output>+----------+--------+-----------+-------------+
|      date|    time|transaction|         item|
+----------+--------+-----------+-------------+
|2016-10-30|09:58:11|          1|        Bread|
|2016-10-30|10:05:34|          2| Scandinavian|
|2016-10-30|10:05:34|          2| Scandinavian|
|2016-10-30|10:07:57|          3|Hot chocolate|
|2016-10-30|10:07:57|          3|          Jam|
|2016-10-30|10:07:57|          3|      Cookies|
|2016-10-30|10:08:41|          4|       Muffin|
|2016-10-30|10:13:03|          5|       Coffee|
|2016-10-30|10:13:03|          5|       Pastry|
|2016-10-30|10:13:03|          5|        Bread|
+----------+--------+-----------+-------------+
only showing top 10 rows<jupyter_text>Transform Data with Spark SQLTransform the DataFrame's bakery data using Spark SQL<jupyter_code>df_bakery1.createOrReplaceTempView("bakery_table_tmp1")

df_bakery2 = spark.sql("SELECT date, transaction, item " +
                       "FROM bakery_table_tmp1 " +
                       "WHERE item NOT LIKE 'NONE'" +
                       "ORDER BY transaction")
df_bakery2.show(5)
df_bakery2.count()
df_bakery2.createOrReplaceTempView("bakery_table_tmp2")

df_bakery3 = spark.sql("SELECT date, count(*) as count " +
                       "FROM bakery_table_tmp2 " +
                       "WHERE date >= '2017-01-01' " +
                       "GROUP BY date " +
                       "ORDER BY date")
df_bakery3.show(5)
df_bakery3.count()<jupyter_output>+----------+-----+
|      date|count|
+----------+-----+
|2017-01-01|    1|
|2017-01-03|   87|
|2017-01-04|   76|
|2017-01-05|   95|
|2017-01-06|   84|
+----------+-----+
only showing top 5 rows<jupyter_text>Use Plotly to Visualize DataUse [Plotly](https://plot.ly/python/) to create a chart showing bakery items sold over time  Demostrates linear fit and data smoothing:* [Plotly Python Open Source Graphing Library](https://plot.ly/python/)* [Smoothing in Python](https://plot.ly/python/smoothing/)* [Linear Fit in Python](https://plot.ly/python/linear-fits/)<jupyter_code>! pip install plotly chart-studio --upgrade --quiet
import chart_studio.tools
import chart_studio.plotly as py
import plotly.graph_objs as go
from numpy import arange
from scipy import stats, signal
import warnings
warnings.filterwarnings('ignore')
# *** UPDATE WITH YOUR CREDENTIALS FOR PLOTLY ***
chart_studio.tools.set_credentials_file(username='username_goes_here', api_key='api_key_goes_here')
df_bakery4 = df_bakery3.toPandas()

# Generated linear fit
xi = arange(0, len(df_bakery4.index))
slope, intercept, r_value, p_value, std_err = stats.linregress(xi, df_bakery4['count'])
line = slope * xi + intercept

layout = dict(title='2017 Bakery Sales',
              xaxis=dict(
                  title='Month',
                  showgrid=True,
                  zeroline=True,
                  showline=True,
                  ticks='outside',
                  tickangle=45,
                  showticklabels=True),
              yaxis=dict(
                  title='Items Sold/Day',
                  showgrid=True,
                  zeroline=True,
                  showline=True,
                  ticks='outside',
                  showticklabels=True))

trace1 = go.Bar(x=df_bakery4['date'], y=df_bakery4['count'], name='Items Sold')
trace2 = go.Scatter(x=df_bakery4['date'], y=line, mode='lines', name='Linear Fit')
trace3 = go.Scatter(x=df_bakery4['date'], y=signal.savgol_filter(df_bakery4['count'], 53, 3),
                    mode='lines', name='Savitzky-Golay')
data = [trace1, trace2, trace3]
fig = dict(data=data, layout=layout)
py.iplot(fig, filename='jupyter-basic_bar.html')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.164380714879468, 'fraction_numerical': 0.055070656691604325, 'mean_word_length': 2.6052434456928837, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 17, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '478031', 'license': 'MIT', 'n_tokens_mistral': 1794, 'n_tokens_neox': 1588, 'n_words': 325, 'path': 'work/05_pyspark_demo_notebook.ipynb', 'repo_name': 'Richaro10/pyspark-setup-demo', 'size': '4822'}