<jupyter_start><jupyter_text>Accessing Data From S3This example shows how to configure a JupyterLab docker image to access data from AWS S3. Build a Docker Image with AWS Related JARsFirst, we need to build a docker image that includes the missing jars files needed for accessing S3. You can also add the jars using a volume mount, and then include code in your notebook to update the `PYSPARK_SUBMIT_ARGS` to include the jars from their location within the docker image. I felt like baking the jars into the docker image was a little easier that having to run a code cell to update the `PYSPARK_SUBMIT_ARGS`.This example is using Spark 3.0.1 with Hadoop 3.2, and the files that we're adding are:* aws-java-sdk-bundle-1.11.950.jar* hadoop-aws-3.2.0.jar* jets3t-0.9.4.jarHere is an example Dockerfile to use:```FROM jupyter/pyspark-notebook:8ea7abc5b7bcUSER rootENV PYSPARK_SUBMIT_ARGS '--packages com.amazonaws:aws-java-sdk:1.11.950,org.apache.hadoop:hadoop-aws:3.2.0,net.java.dev.jets3t:jets3t:0.9.4 pyspark-shell' Download missing jars Get AWS SDK JARRUN (cd /usr/local/spark/jars && curl -O https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.950/aws-java-sdk-bundle-1.11.950.jar) Get Hadoop-AWS JarRUN (cd /usr/local/spark/jars && curl -O https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.2.0/hadoop-aws-3.2.0.jar) Get jets3t JARRUN (cd /usr/local/spark/jars && curl -O https://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.9.4/jets3t-0.9.4.jar)USER $NB_UID``` Run the Docker Container and Pass in AWS CredentialsThis example is assuming that you have appropriate credentials saved in $HOME/.aws/credentials, and have jq installed.Fetch temporary credentials from AWS and run the docker container with the credentials and session token passed in as environment variables:```bashcreds_json=$(aws --profile default --region us-west-2 sts get-session-token)docker run -d --name jupyter --rm -p 8888:8888 \  -e AWS_ACCESS_KEY_ID=$(echo "$creds_json" | jq -r .Credentials.AccessKeyId) \  -e AWS_SECRET_ACCESS_KEY=$(echo "$creds_json" | jq -r .Credentials.SecretAccessKey) \  -e AWS_SESSION_TOKEN=$(echo "$creds_json" | jq -r .Credentials.SessionToken) \  jupyter-docker:yourtag jupyter lab --LabApp.token ''```  Configure Spark<jupyter_code>from pyspark.sql import SparkSession
import logging
logging.getLogger().setLevel(logging.DEBUG)<jupyter_output><empty_output><jupyter_text>Set the SparkSession Thread Count and MemoryIf you have JupyterLab running in the cloud, and you can afford to run enough instances where you're not overly concerned with cost, then don't worry about this section. If you are running JupyterLab on a single machine (for example, a laptop with limited resources), and the amount of data you want to process is more than you have available on the machine, then you might want to be thoughtful about how you initialize the SparkSession. If the single machine (perhaps your home laptop) use case sounds like you, then this is what I considered when configuring the SparkSession.I have 8 cores and 16GB of memory available on my laptop, and I configured Docker to use 4 cores and up to 3GB of memory. Things to consider if the Spark cluster is on a constrained system:* How much memory do you have available for your Spark job?    > If you don't have much memory available, then consider reading the [Spark Memory Tuning Guide](https://spark.apache.org/docs/latest/tuning.htmlmemory-tuning). There are great suggestions for everything from changing the default serializer to being aware of the impacts of using broadcast variables.* How much data do you plan to process?     > You also might want to be aware of the format that your source data is in. [Here is a nice article comparing CSV, JSON, and Parquet](https://www.linkedin.com/pulse/spark-file-format-showdown-csv-vs-json-parquet-garren-staubli/). If your data is in JSON, but you want to process the data as Parquet, then consider creating a job to convert the data to Parquet before using the data in your processing jobs.<jupyter_code>MAX_MEMORY = "2g"

spark = SparkSession.builder \
    .master("local[4]") \
    .appName("Covid19TimeSeries") \
    .config("spark.executor.memory", MAX_MEMORY) \
    .config("spark.driver.memory", MAX_MEMORY) \
    .config("fs.s3a.path.style.access", True) \
    .config("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider") \
    .config("fs.s3a.endpoint", "s3.us-west-2.amazonaws.com") \
    .config("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
    .config("com.amazonaws.services.s3.enableV4", True) \
    .config("spark.driver.extraJavaOptions", "-Dcom.amazonaws.services.s3.enableV4=true") \
    .getOrCreate()<jupyter_output><empty_output><jupyter_text>Read Data From S3At this point you should be able to read data in from S3.<jupyter_code>s3path = "s3a://dev-leewallen-spark/covid-19-time-series/parquet/covid-19.parquet"
parquetDF = spark.read.parquet(s3path)
from pyspark.sql.functions import col 
parquetDF.show(5)
usConfirmed = parquetDF.filter((col('`Country/Region`') == "US"))
# import chart_studio.plotly as py
# import plotly.graph_objects as go
# from  plotly.offline import plot
import pandas as pd
import matplotlib.pyplot as plt
import requests
requests.packages.urllib3.disable_warnings()
usConfirmed.show(5)
usConfirmed.printSchema()
from pyspark.sql.types import DateType

usConfirmed = usConfirmed.withColumn("DateTS",usConfirmed["Date"].cast(DateType()))
usConfirmed.show()
usConfirmed.printSchema()
usPandas = usConfirmed.toPandas()
usPandas<jupyter_output><empty_output><jupyter_text>Pandas Plot Related Settings<jupyter_code>pd.options.plotting.matplotlib.register_converters = True
plt.close("all")<jupyter_output><empty_output><jupyter_text>Make the Plot InteractiveMake the plot resizeable, and provide an interface so you can save your plot.<jupyter_code>%matplotlib widget
usPandas.plot.bar(stacked=True)
usPandas.plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10075062552126772, 'fraction_numerical': 0.019349457881567972, 'mean_word_length': 4.930761622156281, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '504707', 'license': 'MIT', 'n_tokens_mistral': 1991, 'n_tokens_neox': 1897, 'n_words': 611, 'path': 'notebooks/AwsS3AccessTemporaryCredentials.ipynb', 'repo_name': 'leewallen/jupyter-with-aws-s3', 'size': '5995'}