<jupyter_start><jupyter_text>Load the data<jupyter_code>parse_date = lambda date: pd.datetime.strptime(date, '%Y-%m')
data = pd.read_csv('data/AirPassengers.csv', parse_dates=['Month'], index_col='Month', date_parser=parse_date)
data.head()
data.dtypes
ts = data['#Passengers']
ts.head()
from datetime import datetime
ts[datetime(1949,1,1)]
plt.plot(ts)<jupyter_output>/Users/pawel/PycharmProjects/data-science/env/lib/python3.6/site-packages/pandas/plotting/_converter.py:129: FutureWarning: Using an implicitly registered datetime converter for a matplotlib plotting method. The converter was registered by pandas on import. Future versions of pandas will require you to explicitly register matplotlib converters.

To register the converters:
	>>> from pandas.plotting import register_matplotlib_converters
	>>> register_matplotlib_converters()
  warnings.warn(msg, FutureWarning)<jupyter_text>Check Stationarity of Time Series- *Plot Rolling Statistics*: this is just a visual technique- *Dickey-Fuller Test*: this is a statistical test. Null hypothesis is that the TS is non-stationary. If the test statistic is less than the critical value, we reject the null hypothesis and say that the time series is stationary.<jupyter_code>from statsmodels.tsa.stattools import adfuller

def test_stationarity(timeseries):
    # calculate rolling statistics
    rolmean = timeseries.rolling(window=12).mean()
    rolstd = timeseries.rolling(window=12).std()

    # plot rolling statistics
    orig = plt.plot(timeseries, color='C0',label='Original')
    mean = plt.plot(rolmean, color='C1', label='Rolling Mean')
    std = plt.plot(rolstd, color='C2', label = 'Rolling Std')
    plt.legend(loc='best')
    plt.title('Rolling Mean & Standard Deviation')
    plt.show(block=False)
    
    #Perform Dickey-Fuller test:
    print('Results of Dickey-Fuller Test:')
    dftest = adfuller(timeseries, autolag='AIC')
    dfoutput = pd.Series(dftest[0:4], index=['Test Statistic','p-value','#Lags Used','Number of Observations Used'])
    for key,value in dftest[4].items():
        dfoutput['Critical Value (%s)'%key] = value
    print(dfoutput)
    
test_stationarity(ts)<jupyter_output><empty_output><jupyter_text>How to make a Time Series stationary?- *Trend* – varying mean over time- *Seasonality* – variations at specific time-frames  Estimating and Eliminating Trend<jupyter_code>ts_log = np.log(ts)
plt.plot(ts_log)
ts_log_rolling_mean = ts_log.rolling(12).mean()
ts_log_diff = ts_log - ts_log_rolling_mean
ts_log_diff.head(12)
ts_log_diff.dropna(inplace=True)
test_stationarity(ts_log_diff)
ts_log_ewm = ts_log.ewm(halflife=12).mean()
plt.plot(ts_log)
plt.plot(ts_log_ewm)
ts_log_ewm_diff = ts_log - ts_log_ewm
test_stationarity(ts_log_ewm_diff)<jupyter_output><empty_output><jupyter_text>Eliminating Trend and Seasonality  Differencing<jupyter_code>ts_log_diff = ts_log - ts_log.shift()
plt.plot(ts_log_diff)
ts_log_diff.dropna(inplace=True)
test_stationarity(ts_log_diff)<jupyter_output><empty_output><jupyter_text>Decomposing<jupyter_code>from statsmodels.tsa.seasonal import seasonal_decompose
decomposition = seasonal_decompose(ts_log)

trend = decomposition.trend
seasonal = decomposition.seasonal
residual = decomposition.resid

plt.plot(ts_log, label='Original')
plt.plot(trend, label='Trend')
plt.plot(seasonal,label='Seasonality')
plt.plot(residual, label='Residuals')
plt.legend(loc='best')
plt.tight_layout()
ts_log_decompose = residual
ts_log_decompose.dropna(inplace=True)
test_stationarity(ts_log_decompose)<jupyter_output><empty_output><jupyter_text>Forecasting Time Series - *ACF* - Autocorrelation Function is a measure of the correlation between the TS with a lagged version of itself.- *PACF* - Partial Autocorrelation Function measures the correlation between the TS with a lagged version of itself but after eliminating variations already explained by the intervening comparisons.<jupyter_code>from statsmodels.tsa.stattools import acf, pacf

lag_acf = acf(ts_log_diff, nlags = 20)
lag_pacf = pacf(ts_log_diff, nlags = 20, method= 'ols')

#Plot ACF
plt.subplot(121)
plt.plot(lag_acf, linewidth = 2)
plt.axhline(y = 0, linestyle = '--', color = 'gray')
plt.axhline(y=-1.96/np.sqrt(len(ts_log_diff)), linestyle='--', color='gray')
plt.axhline(y=1.96/np.sqrt(len(ts_log_diff)), linestyle='--', color='gray')
plt.title('Autocorrelation Function') 

#Plot PACF
plt.subplot(122)
plt.plot(lag_pacf, linewidth = 2)
plt.axhline(y=0, linestyle='--', color='gray')
plt.axhline(y=-1.96/np.sqrt(len(ts_log_diff)), linestyle='--', color='gray')
plt.axhline(y=1.96/np.sqrt(len(ts_log_diff)), linestyle='--', color='gray')
plt.title('Partial Autocorrelation Function')
            
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>AR Model<jupyter_code>from statsmodels.tsa.arima_model import ARIMA
model = ARIMA(ts_log, order=(2, 1, 0))
results_AR = model.fit(disp=-1)  
plt.plot(ts_log_diff)
plt.plot(results_AR.fittedvalues)
plt.title('RSS: {}'.format(sum((results_AR.fittedvalues-ts_log_diff)**2)))<jupyter_output>/Users/pawel/PycharmProjects/data-science/env/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py:171: ValueWarning: No frequency information was provided, so inferred frequency MS will be used.
  % freq, ValueWarning)
/Users/pawel/PycharmProjects/data-science/env/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py:191: FutureWarning: Creating a DatetimeIndex by passing range endpoints is deprecated.  Use `pandas.date_range` instead.
  start=index[0], end=index[-1], freq=freq)
/Users/pawel/PycharmProjects/data-science/env/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py:171: ValueWarning: No frequency information was provided, so inferred frequency MS will be used.
  % freq, ValueWarning)<jupyter_text>MA Model<jupyter_code>model = ARIMA(ts_log, order=(0, 1, 2))  
results_MA = model.fit(disp=-1)  
plt.plot(ts_log_diff)
plt.plot(results_MA.fittedvalues)
plt.title('RSS: {}'.format(sum((results_MA.fittedvalues-ts_log_diff)**2)))<jupyter_output>/Users/pawel/PycharmProjects/data-science/env/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py:171: ValueWarning: No frequency information was provided, so inferred frequency MS will be used.
  % freq, ValueWarning)
/Users/pawel/PycharmProjects/data-science/env/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py:171: ValueWarning: No frequency information was provided, so inferred frequency MS will be used.
  % freq, ValueWarning)<jupyter_text>Combined Model<jupyter_code>model = ARIMA(ts_log, order=(2, 1, 2))  
results_ARIMA = model.fit(disp=-1)
plt.plot(ts_log_diff)
plt.plot(results_ARIMA.fittedvalues)
plt.title('RSS: {}'.format(sum((results_ARIMA.fittedvalues-ts_log_diff)**2)))
predictions_ARIMA_diff = pd.Series(results_ARIMA.fittedvalues, copy=True)
predictions_ARIMA_diff.head()
predictions_ARIMA_diff_cumsum = predictions_ARIMA_diff.cumsum()
predictions_ARIMA_diff_cumsum.head()
predictions_ARIMA_log = pd.Series(ts_log.ix[0], index=ts_log.index)
predictions_ARIMA_log = predictions_ARIMA_log.add(predictions_ARIMA_diff_cumsum,fill_value=0)
predictions_ARIMA_log.head()
predictions_ARIMA = np.exp(predictions_ARIMA_log)
plt.plot(ts)
plt.plot(predictions_ARIMA)
plt.title('RMSE: {}'.format(np.sqrt(sum((predictions_ARIMA-ts)**2)/len(ts))))<jupyter_output><empty_output><jupyter_text>Predicting 5 years ahead<jupyter_code>plt.figure(figsize=(13,8))
results_ARIMA.plot_predict(1,204)
plt.legend(loc = 'upper left')
plt.show()<jupyter_output>/Users/pawel/PycharmProjects/data-science/env/lib/python3.6/site-packages/statsmodels/tsa/base/tsa_model.py:320: FutureWarning: Creating a DatetimeIndex by passing range endpoints is deprecated.  Use `pandas.date_range` instead.
  freq=base_index.freq)
{'fraction_non_alphanumeric': 0.12637149864463662, 'fraction_numerical': 0.014328127016909772, 'mean_word_length': 6.8183652875882945, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '111878', 'license': 'MIT', 'n_tokens_mistral': 2905, 'n_tokens_neox': 2757, 'n_words': 508, 'path': 'Air Passengers/solution.ipynb', 'repo_name': 'ppisarski/data-science', 'size': '7747'}