<jupyter_start><jupyter_text>Apple Stock  Introduction:We are going to use Apple's stock price. Step 1. Import the necessary libraries<jupyter_code>import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('apple').getOrCreate()
spark
from pyspark import SparkFiles<jupyter_output><empty_output><jupyter_text>Step 2. Import the dataset from this [address](https://raw.githubusercontent.com/guipsamora/pandas_exercises/master/09_Time_Series/Apple_Stock/appl_1980_2014.csv)  Step 3. Assign it to a variable apple<jupyter_code>url = "https://raw.githubusercontent.com/guipsamora/pandas_exercises/master/09_Time_Series/Apple_Stock/appl_1980_2014.csv"

spark.sparkContext.addFile(url)

apple = spark.read.csv(SparkFiles.get("appl_1980_2014.csv"), header=True, inferSchema=True, sep=',')
apple.show(5)<jupyter_output>+----------+-----+-----+-----+-----+--------+---------+
|      Date| Open| High|  Low|Close|  Volume|Adj Close|
+----------+-----+-----+-----+-----+--------+---------+
|2014-07-08|96.27| 96.8|93.92|95.35|65130000|    95.35|
|2014-07-07|94.14|95.99| 94.1|95.97|56305400|    95.97|
|2014-07-03|93.67| 94.1| 93.2|94.03|22891800|    94.03|
|2014-07-02|93.87|94.06|93.09|93.48|28420900|    93.48|
|2014-07-01|93.52|94.07|93.13|93.52|38170200|    93.52|
+----------+-----+-----+-----+-----+--------+---------+
only showing top 5 rows<jupyter_text>Step 4.  Check out the type of the columns<jupyter_code>apple.printSchema()<jupyter_output>root
 |-- Date: string (nullable = true)
 |-- Open: double (nullable = true)
 |-- High: double (nullable = true)
 |-- Low: double (nullable = true)
 |-- Close: double (nullable = true)
 |-- Volume: integer (nullable = true)
 |-- Adj Close: double (nullable = true)<jupyter_text>Step 5. Transform the Date column as a datetime type<jupyter_code>from pyspark.sql.types import *
apple_date = apple.withColumn('Date', apple.Date.cast(DateType()))
apple_date.printSchema()
apple_date.show(5)<jupyter_output>+----------+-----+-----+-----+-----+--------+---------+
|      Date| Open| High|  Low|Close|  Volume|Adj Close|
+----------+-----+-----+-----+-----+--------+---------+
|2014-07-08|96.27| 96.8|93.92|95.35|65130000|    95.35|
|2014-07-07|94.14|95.99| 94.1|95.97|56305400|    95.97|
|2014-07-03|93.67| 94.1| 93.2|94.03|22891800|    94.03|
|2014-07-02|93.87|94.06|93.09|93.48|28420900|    93.48|
|2014-07-01|93.52|94.07|93.13|93.52|38170200|    93.52|
+----------+-----+-----+-----+-----+--------+---------+
only showing top 5 rows<jupyter_text>Step 6.  Set the date as the index  Step 7.  Is there any duplicate dates?<jupyter_code>apple_date.groupBy('Date').count().filter('count>1').show()<jupyter_output>+----+-----+
|Date|count|
+----+-----+
+----+-----+<jupyter_text>Step 8.  Ops...it seems the index is from the most recent date. Make the first entry the oldest date.<jupyter_code>apple_date.orderBy('Date').show(5)<jupyter_output>+----------+-----+-----+-----+-----+---------+---------+
|      Date| Open| High|  Low|Close|   Volume|Adj Close|
+----------+-----+-----+-----+-----+---------+---------+
|1980-12-12|28.75|28.87|28.75|28.75|117258400|     0.45|
|1980-12-15|27.38|27.38|27.25|27.25| 43971200|     0.42|
|1980-12-16|25.37|25.37|25.25|25.25| 26432000|     0.39|
|1980-12-17|25.87| 26.0|25.87|25.87| 21610400|      0.4|
|1980-12-18|26.63|26.75|26.63|26.63| 18362400|     0.41|
+----------+-----+-----+-----+-----+---------+---------+
only showing top 5 rows<jupyter_text>Step 9. Get the last business day of each month<jupyter_code>#trying to find an answer<jupyter_output><empty_output><jupyter_text>Step 10.  What is the difference in days between the first day and the oldest<jupyter_code>import pyspark.sql.functions as F
first = apple_date.orderBy('Date').head(1)[0][0]
print(first)
last = apple_date.orderBy('Date').tail(1)[0][0]
print(last)
df = spark.createDataFrame([(first, last)], ['first', 'last'])
df.show()
F.datediff()
df.select(F.datediff(df["last"], df["first"]).alias('diff')).head(1)[0][0]<jupyter_output><empty_output><jupyter_text>Step 11.  How many months in the data we have?<jupyter_code>df.select(F.months_between(df["last"], df["first"]).alias('diff')).head(1)[0][0]<jupyter_output><empty_output><jupyter_text>Step 12. Plot the 'Adj Close' value. Set the size of the figure to 13.5 x 9 inches<jupyter_code>import pandas as pd
import numpy as np

# visualization
import matplotlib.pyplot as plt

%matplotlib inline
url = 'https://raw.githubusercontent.com/guipsamora/pandas_exercises/master/09_Time_Series/Apple_Stock/appl_1980_2014.csv'
apple_pandas = pd.read_csv(url)

apple_pandas.head()
apple_pandas.sort_values(by='Date',inplace=True)
apple_pandas.set_index('Date',inplace=True)
apple_pandas.head()
# makes the plot and assign it to a variable
appl_open = apple_pandas['Adj Close'].plot(title = "Apple Stock")

# changes the size of the graph
fig = appl_open.get_figure()
fig.set_size_inches(13.5, 9)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.24792887452010506, 'fraction_numerical': 0.12244897959183673, 'mean_word_length': 3.326923076923077, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '507140', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2276, 'n_tokens_neox': 1913, 'n_words': 332, 'path': '09_Time_Series/Apple_Stock/PySparkSolution.ipynb', 'repo_name': 'dhruv-agg/pyspark_practice', 'size': '4949'}