<jupyter_start><jupyter_text>Transposing data<jupyter_code>df = df.T
country = df.iloc[0].to_list()
df.columns = country
df = df.drop('Country/Region')
print(df)<jupyter_output>Afghanistan Albania Algeria Andorra Angola Antigua and Barbuda  \
1/22/20           0       0       0       0      0                   0   
1/23/20           0       0       0       0      0                   0   
1/24/20           0       0       0       0      0                   0   
1/25/20           0       0       0       0      0                   0   
1/26/20           0       0       0       0      0                   0   
...             ...     ...     ...     ...    ...                 ...   
6/14/20       24766    1521   10919     853    140                  26   
6/15/20       25527    1590   11031     853    142                  26   
6/16/20       26310    1672   11147     854    148                  26   
6/17/20       26874    1722   11268     854    155                  26   
6/18/20       27532    1788   11385     855    166                  26   

        Argentina Armenia Australia Austria  ... United Kingdom Uruguay  \
1/22/20         0       0         0 [...]<jupyter_text>converting transposed data into dataframe<jupyter_code>df = pd.DataFrame(df, dtype='float64')
df.head()<jupyter_output><empty_output><jupyter_text>converting index to 'Datatime index' for time series analysis<jupyter_code>df.index = pd.to_datetime(df.index)
df.index<jupyter_output><empty_output><jupyter_text>datetime frequency to 'Daily' data<jupyter_code>df.index.freq = 'D'
df.index
df.info()
df.index
df.dropna(inplace=True)
df.shape
df['India'].plot(figsize = (12,6))
len(df)<jupyter_output><empty_output><jupyter_text>Splitting data to train & test set<jupyter_code>train = df.iloc[:120]
test = df.iloc[120:]
len(test)
from statsmodels.tsa.stattools import adfuller

import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Augmented Dickey-Fuller TestLet's make a custom function to perform Augmented Dickey-Fuller test and check whether the data is stationary or not<jupyter_code>def adf_test(series,title=''):
    """
    Pass in a time series and an optional title, returns an ADF report
    """
    print(f'Augmented Dickey-Fuller Test: {title}')
    result = adfuller(series.dropna(),autolag='AIC') # .dropna() handles differenced data
    
    labels = ['ADF test statistic','p-value','# lags used','# observations']
    out = pd.Series(result[0:4],index=labels)

    for key,val in result[4].items():
        out[f'critical value ({key})']=val
        
    print(out.to_string())          # .to_string() removes the line "dtype: float64"
    
    if result[1] <= 0.05:
        print("Strong evidence against the null hypothesis")
        print("Reject the null hypothesis")
        print("Data has no unit root and is stationary")
    else:
        print("Weak evidence against the null hypothesis")
        print("Fail to reject the null hypothesis")
        print("Data has a unit root and is non-stationary")
adf_test(df['India'])<jupyter_output>Augmented Dickey-Fuller Test: 
ADF test statistic       -4.658241
p-value                   0.000101
# lags used              14.000000
# observations          134.000000
critical value (1%)      -3.480119
critical value (5%)      -2.883362
critical value (10%)     -2.578407
Strong evidence against the null hypothesis
Reject the null hypothesis
Data has no unit root and is stationary<jupyter_text>*As we can see that the data is stationary, we can apply ARIMA model to this data.*<jupyter_code>from pmdarima import auto_arima
from statsmodels.tsa.arima_model import ARIMA,ARIMAResults<jupyter_output><empty_output><jupyter_text>auto_arimaIn order to fit ARIMA model we have to find suitable p, d & q term. In order to do that we will use 'auto_arima' funtion from pmdarima (pyramid arima) library. It is very similar to 'grid search' we use in ML to find out suitable hyperparameters. Here, p, d, q terms are also like hyperparameters.<jupyter_code>stepwise_fit = auto_arima(df['India'], start_p=0, start_q=0,
                          max_p=6, max_q=3, m=1,
                          seasonal=False,
                          d=None, trace=True,
                          error_action='ignore',   # we don't want to know if an order does not work
                          suppress_warnings=True,  # we don't want convergence warnings
                          stepwise=True)           # set to stepwise

stepwise_fit.summary()<jupyter_output>Performing stepwise search to minimize aic
Fit ARIMA(0,2,0)x(0,0,0,0) [intercept=True]; AIC=2246.816, BIC=2252.797, Time=0.025 seconds
Fit ARIMA(1,2,0)x(0,0,0,0) [intercept=True]; AIC=2233.606, BIC=2242.577, Time=0.071 seconds
Fit ARIMA(0,2,1)x(0,0,0,0) [intercept=True]; AIC=2228.860, BIC=2237.831, Time=0.302 seconds
Fit ARIMA(0,2,0)x(0,0,0,0) [intercept=False]; AIC=2249.803, BIC=2252.793, Time=0.015 seconds
Fit ARIMA(1,2,1)x(0,0,0,0) [intercept=True]; AIC=2230.814, BIC=2242.776, Time=0.446 seconds
Fit ARIMA(0,2,2)x(0,0,0,0) [intercept=True]; AIC=2230.918, BIC=2242.879, Time=0.288 seconds
Fit ARIMA(1,2,2)x(0,0,0,0) [intercept=True]; AIC=2226.309, BIC=2241.261, Time=0.561 seconds
Near non-invertible roots for order (1, 2, 2)(0, 0, 0, 0); setting score to inf (at least one inverse root too close to the border of the unit circle: 1.000)
Total fit time: 1.729 seconds<jupyter_text>Let's fit the ARIMA model with obtained order for the model<jupyter_code>model = ARIMA(train['India'],order=(1,2,2))
results = model.fit()
results.summary()
# Obtain predicted values
start=len(train)
end=len(train)+len(test)-1
predictions = results.predict(start=start, end=end, dynamic=False, typ='levels').rename('ARIMA(1,2,2) Predictions')
# Compare predictions to expected values
for i in range(len(predictions)):
    print(f"predicted={predictions[i]:<11.10}, expected={test['India'][i]}")
from sklearn.metrics import mean_squared_error

error = mean_squared_error(test['India'], predictions)
print(f'ARIMA(1,2,2) MSE Error: {error:11.10}')
from statsmodels.tools.eval_measures import rmse

error = rmse(test['India'], predictions)
print(f'ARIMA(1,2,2) RMSE Error: {error:11.10}')
model = ARIMA(df['India'],order=(1,2,2))
results = model.fit()
fcast = results.predict(len(df),len(df)+1,typ='levels').rename('ARIMA(1,2,2) Forecast')<jupyter_output><empty_output><jupyter_text>Predicted values in future<jupyter_code>fcast
# Plot predictions against known values

ax = df['India'].plot(legend=True,figsize=(12,6))
fcast.plot(legend=True)
ax.autoscale(axis='x',tight=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11979480989740494, 'fraction_numerical': 0.07619191309595655, 'mean_word_length': 2.9481834425253126, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '396746', 'license': 'Apache-2.0', 'n_tokens_mistral': 2546, 'n_tokens_neox': 2266, 'n_words': 573, 'path': 'single_country_pred.ipynb', 'repo_name': 'siddheshgunjal/Covid-19-predictions', 'size': '6628'}