<jupyter_start><jupyter_text>Loading Data from CSV file * The Pandas library in Python provides excellent, built-in support for time series data.* Pandas represents time series datasets as a **Series**. * A Series is a one-dimensional array with a time label for each row.* A Dataframe is a collection of series<jupyter_code>import pandas as pd
import numpy as np
# Download csv file from resources and put it in working directory
dataframe = pd.read_csv('daily-total-female-births-CA.csv', header=0)<jupyter_output><empty_output><jupyter_text>First five records<jupyter_code>dataframe.head()<jupyter_output><empty_output><jupyter_text>Data Type<jupyter_code>dataframe['date'].dtype<jupyter_output><empty_output><jupyter_text>Loading data with parse_dates<jupyter_code>df2 = pd.read_csv('daily-total-female-births-CA.csv', header=0 , parse_dates=[0])<jupyter_output><empty_output><jupyter_text>First five records<jupyter_code>df2.head()<jupyter_output><empty_output><jupyter_text>Data Type<jupyter_code>df2['date'].dtype<jupyter_output><empty_output><jupyter_text>Notedateparse = lambda x: pd.datetime.strptime(x, '%Y-%m-%d %H:%M:%S')df = pd.read_csv(.........,parse_dates=[0], date_parser=dateparse)https://docs.python.org/2/library/datetime.htmlstrftime-and-strptime-behavior  Loading Data as a series<jupyter_code>series = pd.read_csv('daily-total-female-births-CA.csv', header=0, parse_dates=[0], index_col=0, squeeze=True)<jupyter_output><empty_output><jupyter_text>First 5 records<jupyter_code>series.head()<jupyter_output><empty_output><jupyter_text>Exploring Time Series Data  Size<jupyter_code>series.shape
df2.shape<jupyter_output><empty_output><jupyter_text>Querying by time<jupyter_code>print(series['1959-01'])
df2[(df2['date'] > '1959-01-01') & (df2['date'] <= '1959-01-21')]<jupyter_output><empty_output><jupyter_text>Descriptive Statistics<jupyter_code>series.describe()
df2.describe()<jupyter_output><empty_output><jupyter_text>Feature Engineering * Date time features* Lag features* Window features* Expanding feature  Date time features<jupyter_code>df2.head(5)
features = df2.copy()
features['year'] = df2['date'].dt.year
features['month'] = df2['date'].dt.month
features['day'] = df2['date'].dt.day
features.head(5)<jupyter_output><empty_output><jupyter_text>For more detail : https://pandas.pydata.org/pandas-docs/stable//reference/series.htmldatetimelike-properties  Lag features To create lag1 feature<jupyter_code>features['lag1'] =  df2['births'].shift(2)
features['lag2'] =  df2['births'].shift(365)
features.head(7)<jupyter_output><empty_output><jupyter_text>For more detail : https://pandas.pydata.org/pandas-docs/stable//reference/api/pandas.Series.shift.htmlpandas.Series.shift  Window features<jupyter_code>features['Roll_mean'] = df2['births'].rolling(window = 2).mean()
features.head(5)
features['Roll_max'] = df2['births'].rolling(window = 3).max()
features.head(5)<jupyter_output><empty_output><jupyter_text>For more detail : https://pandas.pydata.org/pandas-docs/stable//reference/api/pandas.Series.rolling.htmlpandas.Series.rolling  Expanding features<jupyter_code>features['Expand_max'] = df2['births'].expanding().max()
features.head(10)<jupyter_output><empty_output><jupyter_text>For more detail : https://pandas.pydata.org/pandas-docs/stable//reference/api/pandas.Series.expanding.htmlpandas.Series.expanding  Time Series - Data Visualization<jupyter_code>from matplotlib import pyplot as plt
%matplotlib inline
Dataviz_df = df2.copy()
Dataviz_df.head(10)
Dataviz_df['births'].plot()
Dataviz_df.index = Dataviz_df['date'] 
Dataviz_df['births'].plot()<jupyter_output><empty_output><jupyter_text>Zooming In<jupyter_code>Dataviz_df2 = Dataviz_df[(Dataviz_df['date'] > '1959-01-01') & (Dataviz_df['date'] <= '1959-01-10')].copy()
Dataviz_df2
Dataviz_df2['births'].plot()<jupyter_output><empty_output><jupyter_text>Trendline<jupyter_code>import seaborn as sns
sns.regplot(x= df2.index.values, y=df2['births'])
sns.regplot(x= df2.index.values, y=df2['births'], order =2)
miles_df = pd.read_csv('us-airlines-monthly-aircraft-miles-flown.csv', header=0 , parse_dates=[0])
miles_df.head()
miles_df['MilesMM'].plot()
sns.regplot(x= miles_df.index.values, y=miles_df['MilesMM'])<jupyter_output><empty_output><jupyter_text>Removing Seasonality<jupyter_code>miles_df['year'] = miles_df['Month'].dt.year
miles_df.head()
miles_df.groupby('year')['MilesMM'].mean()
miles_df.groupby('year')['MilesMM'].mean().plot()<jupyter_output><empty_output><jupyter_text>Creating Lag plots<jupyter_code>miles_df['lag1'] =  miles_df['MilesMM'].shift(1)
miles_df.head()
sns.scatterplot(x=miles_df['lag1'], y=miles_df['MilesMM'])
from pandas.plotting import lag_plot
lag_plot(miles_df['MilesMM'])<jupyter_output><empty_output><jupyter_text>Autocorrelation Plots<jupyter_code>from pandas.plotting import autocorrelation_plot
autocorrelation_plot(miles_df['MilesMM'])<jupyter_output><empty_output><jupyter_text>Downsampling and Upsampling<jupyter_code>import pandas as pd
miles_df = pd.read_csv('us-airlines-monthly-aircraft-miles-flown.csv', header=0 , parse_dates=[0])
miles_df.head()<jupyter_output><empty_output><jupyter_text>Downsampling<jupyter_code>quarterly_miles_df = miles_df.resample('Q', on='Month').mean()
quarterly_miles_df.head()
yearly_total_miles_df = miles_df.resample('A', on='Month').sum()
yearly_total_miles_df.head()<jupyter_output><empty_output><jupyter_text>| Alias  | Description           ||--------|-----------------------|| B      | Business day          || D      | Calendar day          || W      | Weekly                || M      | Month end             || Q      | Quarter end           || A      | Year end              || BA     | Business year end     || AS     | Year start            || H      | Hourly frequency      || T, min | Minutely frequency    || S      | Secondly frequency    || L, ms  | Millisecond frequency || U, us  | Microsecond frequency || N, ns  | Nanosecond frequency  |  Upsampling<jupyter_code>upsampled_miles_df = miles_df.resample('D', on='Month').mean()
upsampled_miles_df.head(35)
interpolated_miles_df = upsampled_miles_df.interpolate(method='linear')
interpolated_miles_df.head(32)
interpolated_miles_df.plot()
poly_interpolated_miles_df = upsampled_miles_df.interpolate(method='spline', order=2)
poly_interpolated_miles_df.head(35)
poly_interpolated_miles_df.plot()<jupyter_output><empty_output><jupyter_text>| Method  | Description                                               ||---------|-----------------------------------------------------------|| bfill   | Backward fill                                             || count   | Count of values                                           || ffill   | Forward fill                                              || first   | First valid data value                                    || last    | Last valid data value                                     || max     | Maximum data value                                        || mean    | Mean of values in time range                              || median  | Median of values in time range                            || min     | Minimum data value                                        || nunique | Number of unique values                                   || ohlc    | Opening value, highest value, lowest value, closing value || pad     | Same as forward fill                                      || std     | Standard deviation of values                              || sum     | Sum of values                                             || var     | Variance of values                                        |  Decomposing Time Series **Additive Model** y(t) = Level + Trend + Seasonality + Noise**Multiplicative Model** y(t) = Level * Trend * Seasonality * Noise<jupyter_code>from statsmodels.tsa.seasonal import seasonal_decompose
miles_decomp_df = pd.read_csv('us-airlines-monthly-aircraft-miles-flown.csv', header=0 , parse_dates=[0])
miles_decomp_df.head()
miles_decomp_df.index = miles_decomp_df['Month'] 
result = seasonal_decompose(miles_decomp_df['MilesMM'], model='additive')
result.plot()
result2 = seasonal_decompose(miles_decomp_df['MilesMM'], model='multiplicative')
result2.plot()<jupyter_output><empty_output><jupyter_text>Differencing<jupyter_code>miles_df = pd.read_csv('us-airlines-monthly-aircraft-miles-flown.csv', header=0 , parse_dates=[0])
miles_df.head()
miles_df['lag1'] =  miles_df['MilesMM'].shift(1)
miles_df['MilesMM_diff_1'] = miles_df['MilesMM'].diff(periods=1)
miles_df.head()
miles_df.index = miles_df['Month'] 
result_a = seasonal_decompose(miles_df['MilesMM'], model='additive')
result_a.plot()
miles_df.index = miles_df['Month'] 
result_b = seasonal_decompose(miles_df.iloc[1:,3], model='additive')
result_b.plot()
miles_df['MilesMM'].plot()
miles_df['MilesMM_diff_1'].plot() 
miles_df['MilesMM_diff_12'] = miles_df['MilesMM_diff_1'].diff(periods=12)
miles_df['MilesMM_diff_12'].plot() 
result_c = seasonal_decompose(miles_df.iloc[13:,4], model='additive')
result_c.plot()
miles_df.head(50)<jupyter_output><empty_output><jupyter_text>Test-Train Split<jupyter_code>temp_df = pd.read_csv('daily-min-temperatures.csv', header=0 , parse_dates=[0])
temp_df.head()
temp_df.tail()
temp_df.shape
temp_df.shape[0]
train_size = int(temp_df.shape[0]*0.8)
train_size
train =  temp_df[0:train_size]
test =  temp_df[train_size:]
train.shape
test.shape<jupyter_output><empty_output><jupyter_text>Walk Forward Validation ![walkforward2.png](attachment:walkforward2.png)  Persistence Model<jupyter_code>df = pd.read_csv('daily-min-temperatures.csv', header=0 , parse_dates=[0])
df.head()
df['t'] =  df['Temp'].shift(1)
df.head()
train, test = df[1:df.shape[0]-7], df[df.shape[0]-7:]
train.head()
train_X, train_y = train['t'], train['Temp']
test_X, test_y = test['t'], test['Temp']<jupyter_output><empty_output><jupyter_text>walk-forward validation<jupyter_code>predictions = test_X.copy()
print(predictions)
print(test_y)
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(test_y, predictions)
mse
from matplotlib import pyplot
pyplot.plot(test_y)
pyplot.plot(predictions, color='red')<jupyter_output><empty_output><jupyter_text>Autoregression Model<jupyter_code>df = pd.read_csv('daily-min-temperatures.csv', header=0 , parse_dates=[0])
train, test = df.Temp[1:df.shape[0]-7], df.Temp[df.shape[0]-7:]
train.head()
from statsmodels.tsa.ar_model import AR
model = AR(train)
model_fit = model.fit()<jupyter_output><empty_output><jupyter_text>No. of lag Variables<jupyter_code>model_fit.k_ar<jupyter_output><empty_output><jupyter_text>Coef of lag variables<jupyter_code>model_fit.params
predictions = model_fit.predict(start=len(train), end=len(train)+len(test)-1)
predictions
predictions.iloc[0]
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(test_y, predictions)
mse
pyplot.plot(test_y)
pyplot.plot(predictions, color='red')<jupyter_output><empty_output><jupyter_text>Walk Forward<jupyter_code>df = pd.read_csv('daily-min-temperatures.csv', header=0 , parse_dates=[0])
train, test = df.Temp[1:df.shape[0]-7], df.Temp[df.shape[0]-7:]
data = train
predict =[]
for t in test:
    model = AR(data)
    model_fit = model.fit()
    y = model_fit.predict(start=len(data), end=len(train)+len(test)-1)
    print(y.values[0])
    predict.append(y.values[0])
    data = np.append(data, t)
    data = pd.Series(data)
    
predict
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(test.values, predict)
mse
pyplot.plot(test.values)
pyplot.plot(predict, color='red')<jupyter_output><empty_output><jupyter_text>Moving Average Model<jupyter_code>df = pd.read_csv('daily-min-temperatures.csv', header=0 , parse_dates=[0])
df['t'] =  df['Temp'].shift(1)
df['Resid'] = df['Temp'] - df['t']
df.head()
train, test = df.Resid[1:df.shape[0]-7], df.Resid[df.shape[0]-7:]
train.head()
from statsmodels.tsa.ar_model import AR
model = AR(train)
model_fit = model.fit()
model_fit.k_ar
model_fit.params
pred_resid = model_fit.predict(start=len(train), end=len(train)+len(test)-1)
pred_resid
df.t[df.shape[0]-7:]
predictions = df.t[df.shape[0]-7:] + pred_resid
predictions
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(test_y, predictions)
mse
pyplot.plot(test_y)
pyplot.plot(predictions, color='red')<jupyter_output><empty_output><jupyter_text>ARIMA Model<jupyter_code>df = pd.read_csv('shampoo.csv', header=0, parse_dates=[0])
df.head()
df['Sales'].plot()<jupyter_output><empty_output><jupyter_text>Trend is Polynomial.  D =2  Autocorrelation Plot<jupyter_code>from pandas.plotting import autocorrelation_plot
autocorrelation_plot(df['Sales'])<jupyter_output><empty_output><jupyter_text>q = 5   Partial Autocorrelation Graph<jupyter_code>from statsmodels.graphics.tsaplots import plot_pacf
plot_pacf(df['Sales'], lags=15)<jupyter_output><empty_output><jupyter_text>p = 2<jupyter_code>from statsmodels.tsa.arima_model import ARIMA
model = ARIMA(df['Sales'], order=(5,2,2))
model_fit = model.fit()
model_fit.summary()
residuals = model_fit.resid
residuals.plot()
residuals.describe()<jupyter_output><empty_output><jupyter_text>Variations* **ARIMA** -     model = ARIMA(df['Sales'], order=(q,d,p)) * **Autoregression** -     model = ARIMA(df['Sales'], order=(p,d,0)) * **Moving Avaerage Model** -     model = ARIMA(df['Sales'], order=(0,d,q))<jupyter_code>output = model_fit.forecast()
output
model_fit.forecast(5)[0]<jupyter_output><empty_output><jupyter_text>Walk Forward ARIMA<jupyter_code>df = pd.read_csv('shampoo.csv', header=0, parse_dates=[0])
df.head()<jupyter_output><empty_output><jupyter_text>def parser(x):return datetime.strptime('190'+x, '%Y-%m')<jupyter_code>train_size = int(df.shape[0]*0.7)
train, test = df.Sales[0:train_size], df.Sales[train_size:]
test.shape
data = train
predict =[]
for t in test:
    model = ARIMA(data, order=(5,1,0))
    model_fit = model.fit()
    y = model_fit.forecast()
    print(y[0][0])
    predict.append(y[0][0])
    data = np.append(data, t)
    data = pd.Series(data)
predict
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(test.values, predict)
mse<jupyter_output><empty_output><jupyter_text>SARIMA or Seasonal ARIMA 	SARIMA(p,d,q)(P,D,Q)m* p: Trend autoregression order.* d: Trend difference order.* q: Trend moving average order.**Seasonal Elements** -   There are four seasonal elements that are not part of ARIMA that must be configured; they are:  * P: Seasonal autoregressive order.* D: Seasonal difference order.* Q: Seasonal moving average order.* m: The number of time steps for a single seasonal period.<jupyter_code>from statsmodels.tsa.statespace.sarimax import SARIMAX
df = pd.read_csv('us-airlines-monthly-aircraft-miles-flown.csv', header=0 , parse_dates=[0])
df.head()
df.tail()
df.index = df['Month'] 
result_a = seasonal_decompose(df['MilesMM'], model='multiplicative')
result_a.plot()
model = SARIMAX(df['MilesMM'], order=(5,1,3), seasonal_order=(1,1,1,12))
model_fit = model.fit()
residuals = model_fit.resid
residuals.plot()
output = model_fit.forecast()
output
model_fit.forecast(12)
yhat = model_fit.predict()
yhat.head()
pyplot.plot(df['MilesMM'])
pyplot.plot(yhat, color='red')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15067229106248353, 'fraction_numerical': 0.014632217242288427, 'mean_word_length': 5.2286535303776684, 'pattern_counts': {'":': 0, '<': 171, '<?xml version=': 0, '>': 171, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 30, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '213867', 'license': 'MIT', 'n_tokens_mistral': 5653, 'n_tokens_neox': 5443, 'n_words': 834, 'path': 'Forecasting+Resources+(ST+Academy)/Forecasting Resources (ST Academy)/Data + Code/Time Series.ipynb', 'repo_name': 'nishida98/timeSeriesAnalysisAndForecastUsingPython', 'size': '15172'}