<jupyter_start><jupyter_text>Stock Market Analysis of the S&P 500 Index using ARIMA and Seasonal ARIMA for forecastingThis post includes code and notes from [python for finance and trading algorithms udemy course](https://udemy.com/python-for-finance-and-trading-algorithms/) and [python for finance and trading algorithms udemy course notebooks](https://github.com/theoneandonlywoj/Python-for-Financial-Analysis-and-Algorithmic-Trading).<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import pandas_datareader
import datetime
import pandas_datareader.data as web

import statsmodels.api as sm
import quandl

start = datetime.datetime(1960, 1, 1)
end = pd.to_datetime('today')
SP500 = quandl.get("MULTPL/SP500_REAL_PRICE_MONTH",start_date = start,end_date = end)
SP500
df = SP500
df.head()
df.tail()
df.columns = ['Value']
df.head()
df.head()
df.describe().transpose()<jupyter_output><empty_output><jupyter_text>Step 2: Visualize the DataLet's visualize this data with a few methods.<jupyter_code>df.plot()
timeseries = df['Value']
timeseries.rolling(12).mean().plot(label='12 Month Rolling Mean')
timeseries.rolling(12).std().plot(label='12 Month Rolling Std')
timeseries.plot()
plt.legend()
timeseries.rolling(12).mean().plot(label='12 Month Rolling Mean')
timeseries.plot()
plt.legend()<jupyter_output><empty_output><jupyter_text>DecompositionETS decomposition allows us to see the individual parts!<jupyter_code>from statsmodels.tsa.seasonal import seasonal_decompose
decomposition = seasonal_decompose(df['Value'], freq=12)  
fig = plt.figure()  
fig = decomposition.plot()  
fig.set_size_inches(15, 8)<jupyter_output><empty_output><jupyter_text>Testing for Stationarity<jupyter_code>df.head()
from statsmodels.tsa.stattools import adfuller
result = adfuller(df['Value'])
print('Augmented Dickey-Fuller Test:')
labels = ['ADF Test Statistic','p-value','#Lags Used','Number of Observations Used']

for value,label in zip(result,labels):
    print(label+' : '+str(value) )
    
if result[1] <= 0.05:
    print("strong evidence against the null hypothesis, reject the null hypothesis. Data has no unit root and is stationary")
else:
    print("weak evidence against null hypothesis, time series has a unit root, indicating it is non-stationary ")
def adf_check(time_series):
    """
    Pass in a time series, returns ADF report
    """
    result = adfuller(time_series)
    print('Augmented Dickey-Fuller Test:')
    labels = ['ADF Test Statistic','p-value','#Lags Used','Number of Observations Used']

    for value,label in zip(result,labels):
        print(label+' : '+str(value) )
    
    if result[1] <= 0.05:
        print("strong evidence against the null hypothesis, reject the null hypothesis. Data has no unit root and is stationary")
    else:
        print("weak evidence against null hypothesis, time series has a unit root, indicating it is non-stationary ")<jupyter_output><empty_output><jupyter_text>** First Difference **<jupyter_code>df['Value First Difference'] = df['Value'] - df['Value'].shift(1)
adf_check(df['Value First Difference'].dropna())
df['Value First Difference'].plot()<jupyter_output><empty_output><jupyter_text>** Second Difference **<jupyter_code>df['Value Second Difference'] = df['Value First Difference'] - df['Value First Difference'].shift(1)
adf_check(df['Value Second Difference'].dropna())
df['Value Second Difference'].plot()<jupyter_output><empty_output><jupyter_text>** Seasonal Difference **<jupyter_code>df['Seasonal Difference'] = df['Value'] - df['Value'].shift(12)
df['Seasonal Difference'].plot()
adf_check(df['Seasonal Difference'].dropna())<jupyter_output>Augmented Dickey-Fuller Test:
ADF Test Statistic : -5.239903673260254
p-value : 7.284266188346342e-06
#Lags Used : 20
Number of Observations Used : 707
strong evidence against the null hypothesis, reject the null hypothesis. Data has no unit root and is stationary<jupyter_text>** Seasonal First Difference **<jupyter_code>df['Seasonal First Difference'] = df['Value First Difference'] - df['Value First Difference'].shift(12)
df['Seasonal First Difference'].plot()
adf_check(df['Seasonal First Difference'].dropna())
from statsmodels.graphics.tsaplots import plot_acf,plot_pacf
# Check out: https://stackoverflow.com/questions/21788593/statsmodels-duplicate-charts
# https://github.com/statsmodels/statsmodels/issues/1265
fig_first = plot_acf(df["Value First Difference"].dropna())
fig_seasonal_first = plot_acf(df["Seasonal First Difference"].dropna())<jupyter_output><empty_output><jupyter_text>Pandas also has this functionality built in, but only for ACF, not PACF. So I recommend using statsmodels, as ACF and PACF is more core to its functionality than it is to pandas' functionality.<jupyter_code>from pandas.plotting import autocorrelation_plot
autocorrelation_plot(df['Seasonal First Difference'].dropna())<jupyter_output><empty_output><jupyter_text>We can then plot this relationship:<jupyter_code>result = plot_pacf(df["Seasonal First Difference"].dropna())
fig = plt.figure(figsize=(12,8))
ax1 = fig.add_subplot(211)
fig = sm.graphics.tsa.plot_acf(df['Seasonal First Difference'].iloc[13:], lags=40, ax=ax1)
ax2 = fig.add_subplot(212)
fig = sm.graphics.tsa.plot_pacf(df['Seasonal First Difference'].iloc[13:], lags=40, ax=ax2)
from statsmodels.tsa.arima_model import ARIMA
help(ARIMA)
model = sm.tsa.statespace.SARIMAX(df['Value'],order=(0,1,0), seasonal_order=(1,1,1,12))
results = model.fit()
print(results.summary())
results.resid.plot()
results.resid.plot(kind='kde')<jupyter_output><empty_output><jupyter_text>Prediction of Future ValuesFirts we can get an idea of how well our model performs by just predicting for values that we actually already know:<jupyter_code>df['forecast'] = results.predict(start = 1, end= 720, dynamic= True)  
df[['Value','forecast']].plot(figsize=(12,8))<jupyter_output><empty_output><jupyter_text>Forecasting<jupyter_code>df.tail()
from pandas.tseries.offsets import DateOffset
future_dates = [df.index[-1] + DateOffset(months=x) for x in range(0,24) ]
future_dates
future_dates_df = pd.DataFrame(index=future_dates[1:],columns=df.columns)
future_df = pd.concat([df,future_dates_df])
future_df.head()
future_df.tail()
future_df['forecast'] = results.predict(start = 1, end = 720, dynamic= True)  
future_df[['Value', 'forecast']].plot(figsize=(12, 8))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1302717900656045, 'fraction_numerical': 0.023742580443611373, 'mean_word_length': 6.846813725490196, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 48, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '655999', 'license': 'Apache-2.0', 'n_tokens_mistral': 2229, 'n_tokens_neox': 2084, 'n_words': 509, 'path': '_notebooks/2020-10-25-ARIMA-ADF-FORECASTING.ipynb', 'repo_name': 'davidrkearney/Kearney_Data_Science', 'size': '6402'}