<jupyter_start><jupyter_text>the code we wrote for creating the data. https://github.com/anryabrahamyan/Road_Traffic_Prediction/  The following file is the second approach which is averaging our observations for each hour.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
from statsmodels.tsa.stattools import kpss,adfuller
from statsmodels.graphics.tsaplots import plot_acf,plot_pacf
from statsmodels.tsa.arima.model import ARIMA
from statsmodels.tsa.statespace.sarimax import SARIMAX
from statsmodels.tsa.holtwinters import ExponentialSmoothing
import statsmodels.api as sm
import pmdarima as pm
import numpy as np
from datetime import timedelta 
df = pd.read_csv("../dataset/data.csv")
df.datetime = pd.to_datetime(df.datetime,format = "%d/%m/%Y %H:%M:%S")
df.index = df.datetime
df.drop(columns = ['datetime'], inplace = True)
df.head()
df = df.replace("None", np.nan) 
df.iloc[72:144,45] = np.nan
df_cleaned =pd.DataFrame( index=pd.date_range(df.index.min(),df.index.max(),freq='min'))
df_cleaned.index = df_cleaned.index.round(freq='min')
df.index = df.index.round(freq='min')
df_without_na = df[~df['Vehicle_0.1'].isna()]
df_cleaned['Vehicle_0.1'] = np.nan
df_cleaned.loc[df_without_na.index,'Vehicle_0.1']=df_without_na['Vehicle_0.1']
df_cleaned.fillna(method = 'ffill',inplace=True)
df_cleaned['Vehicle_0.1'] = df_cleaned['Vehicle_0.1'].astype(int)
df_cleaned = df_cleaned.groupby([df_cleaned.index.day,df_cleaned.index.hour]).mean().reset_index()
df_cleaned.rename({'level_0':'day','hour':'hour'},inplace=True)
df_cleaned['Vehicle_0.1'].plot()
plt.figure(figsize=(20,8))
plt.plot(df_cleaned["Vehicle_0.1"].astype(float))
plt.show()<jupyter_output><empty_output><jupyter_text>From the plot we can see that there is no trend, however there is some seasonality because the frequency of the cars decreases after 12am.<jupyter_code>X = df_cleaned["Vehicle_0.1"]
result = adfuller(X)
print('ADF Statistic: %f' % result[0])
print('p-value: %f' % result[1])
print('Critical Values:')
for key, value in result[4].items():
    print('\t%s: %.3f' % (key, value))<jupyter_output>ADF Statistic: -4.870671
p-value: 0.000040
Critical Values:
	1%: -3.489
	5%: -2.887
	10%: -2.580<jupyter_text>We can see that p value of ADF statistics is almost 0 which is smaller than 0.05, thus we reject the null hypothesis and conclude that our series is stationary.<jupyter_code>result = kpss(X)
print('KPSS Statistic: %f' % result[0])
print('p-value: %f' % result[1])
print('number of lags: %f' % result[2])

print('Critical Values:')
for key, value in result[3].items():
    print('\t%s: %.3f' % (key, value))<jupyter_output>KPSS Statistic: 0.333687
p-value: 0.100000
number of lags: 13.000000
Critical Values:
	10%: 0.347
	5%: 0.463
	2.5%: 0.574
	1%: 0.739<jupyter_text>We have p-value equal to 0.1, which is larger than 0.05, thus we can say that we have stationary time series.<jupyter_code>plot_acf(X, lags=20)     
plot_pacf(X, lags=20)     
plt.show()<jupyter_output><empty_output><jupyter_text>From the ACF plot we can see that all lags are significant up to lag 3 for the MA process. In PACF we have significant lags until lag 2 and then cuts off, so we will choose lag as 2.<jupyter_code>total_observations = len(X)
train = X[:int(0.8*total_observations)]
test = X[int(0.8*total_observations):]
model = ARIMA(train.astype(float),order=(2, 0, 3)).fit()
print(model.summary())
predictions = model.forecast(len(test))
residuals = test.astype(float) - predictions
plt.figure(figsize=(10,4))
plt.plot(residuals)
plt.axhline(0, linestyle='--', color='red')
plt.title('Residuals from ARIMA Model', fontsize=22)
plt.ylabel('Error', fontsize=17)
Lj_Box = sm.stats.acorr_ljungbox(residuals, lags = 10,return_df = True)
plt.plot(Lj_Box.index, Lj_Box.lb_pvalue)<jupyter_output><empty_output><jupyter_text>The p-values are increasing with the lags, so the model does not capture some of the patterns in the data and the errors are correlated after lag 3(for p-crit = 0.05).<jupyter_code>#WE do not consider the seasonal component because the order of dependence for 1 day is 1440 and it will take forever to fit that model. Further aggregations of the data will help with this problem.
model_auto=pm.auto_arima(
    train,
    test='adf',
    max_p = 12,
    max_d = 1,
    max_q = 12,
    seasonal=True,
    m=24,
    max_P=24,
    max_Q=24,
    max_D=1,
    trace=True,
    error_action='ignore', 
    suppress_warnings=True,
    stepwise=True,
    n_fits=120,
    maxiter=11,
    n_jobs=-1)
print('MSE of the AUTOARIMA model is', np.mean((model_auto.predict(len(test))-test.astype("float"))**2))
print('MSE of the previously chosen model is', np.mean((model.forecast(len(test)) - test.astype("float")) **2))<jupyter_output>MSE of the previously chosen model is 3.6666553749722475<jupyter_text>We can observe that the MSE of the previously chosen model is lower, meaning that it is better if we take into consideration MSEs. This procedure might be incorrect because autoarima minimized the AIC and not the MSE, but the autoarima also included the seasonality of 24 hours, so this is a sign that there is little information in knowing the same number of cars for the previous day.<jupyter_code>model = ARIMA(X.astype(float),order=(2,0,3)).fit()
predictions = model.forecast(24)#24 hours
fig = plt.figure(figsize = (10,10))
plt.plot(X,'blue')
plt.plot(predictions,c='red')
plt.show()
exp_smooth = ExponentialSmoothing(train.astype(float),trend=None, seasonal = 'add',initialization_method='estimated',seasonal_periods=24).fit()<jupyter_output><empty_output><jupyter_text>Car traffic is generally stationary in terms of trend, and the seasonality is additive because the variance is not increasing over time.<jupyter_code>predictions = exp_smooth.forecast(len(test))
print('MSE of the exp_smoothing model is', np.mean((predictions.values-test.astype("float"))**2))<jupyter_output>MSE of the exp_smoothing model is 6.5576617483958195<jupyter_text>The exponential smoothing model is worse than the Arima model from before.<jupyter_code>exp_smooth = ExponentialSmoothing(df_cleaned['Vehicle_0.1'].astype(float),trend=None, seasonal = 'add',initialization_method='estimated',seasonal_periods=24).fit()
ax = plt.subplot()
ax.plot(exp_smooth.forecast(24))
ax.plot(df_cleaned['Vehicle_0.1'].astype(float))
ax.plot(model.forecast(24))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13217035989313217, 'fraction_numerical': 0.03708942322803709, 'mean_word_length': 5.970427163198248, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '16922', 'license': 'Apache-2.0', 'n_tokens_mistral': 2447, 'n_tokens_neox': 2283, 'n_words': 579, 'path': 'predictor/HW4_project_part_with_aggregations.ipynb', 'repo_name': 'anryabrahamyan/Road_Traffic_Prediction', 'size': '6363'}