<jupyter_start><jupyter_text>Bonus Lecture.   Classical Seasonal DecompositionIn this notebook you will learn how to:* Manually use additive and seasonal decomposition i.e. **without** using `statsmodels`* Seasonally adjust your time series to remove seasonal fluctuations.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>The ED arrivals dataset.The dataset we will use represent monthly adult (age > 18) arrivals to an Emergency Department.  The observations are between April 2009 and May 2017.<jupyter_code>ed_month = pd.read_csv('data/ed_mth_ts.csv', index_col='date', parse_dates=True)
ed_month.index.freq = 'MS'

arrival_rate = ed_month['arrivals'] / ed_month.index.days_in_month<jupyter_output><empty_output><jupyter_text>The first thing you should do when exploring a time series is check its length and duration.<jupyter_code>#This tells us how many months are in the ts
arrival_rate.shape<jupyter_output><empty_output><jupyter_text>Breaking a times series up into its trend and seasonal components.To help visualise and understand trend and seasonality in a time series we can use seasonal decomposition.This is a model based approach that breaks the time series into three components.  The basic approach to seasonal decomposition has two forms: additive and multiplicative.   Additive decompositionIf we assume that an observation at time t $Y_t$ is the additive sum of trend $T_t$, seasonality $S_t$ and random error $E_t$.  then we have the following model.$Y_t = T_t + S_t + E_t$We then to make this assumption when the seasonal fluctuations are constant across the time series.  This looks like a reasonable assumption in the case of the ED data. Multiplicative decompositionIf the seasonal fluctuations of the data grow over time then it is best to a multiplicative model. Where an observation at time t $Y_t$ is the product of multiply the trend $T_t$, seasonality $S_t$ and random error $E_t$$Y_t = T_t \cdot  S_t \cdot  E_t$  Manually performing a seasonal decomposition Classical decomposition is straightfoward to implement.  We will work with additive seasonality first. **Additive Seasonality****Step 1: Estimate the trend component**The first thing to do is estimate the trend in the time series.  We do this using a 12 month centered moving average.<jupyter_code>decomp = pd.DataFrame(arrival_rate)
decomp.columns = ['actual']
decomp['trend'] = arrival_rate.rolling(window=12, center=True).mean()
ax = decomp.trend.plot(figsize=(12,4))
arrival_rate.plot(ax=ax)<jupyter_output><empty_output><jupyter_text>**Step 2: Detrend the series**We first need to **de-trend** the time series.  In the additive model we do this by **subtracting** the trend from the actual observations of the time series.$Y_t - T_t = S_t + E_t$<jupyter_code>detrended = decomp['actual'] - decomp['trend']<jupyter_output><empty_output><jupyter_text>When we plot this notice that the upward trend of the data has been removed.<jupyter_code>detrended.plot(figsize=(12,4))<jupyter_output><empty_output><jupyter_text>**Step 3: Calculate the seasonal indicies**Classic decomposition assumes that seasonality is constant year to year.  This means we only need to calculate one value for each month of the year.We work with the detrended series. To calculate the seasonal indicies we group by month and take the average of the detrended series.  This is easy in pandas with a group by operation.<jupyter_code>seasonal_indexes = detrended.groupby(by=detrended.index.month).mean()
seasonal_indexes.plot(figsize=(12,4))<jupyter_output><empty_output><jupyter_text>We then want to **map** the seasonal indicies to the rows in the time series.<jupyter_code>decomp['seasonal'] = decomp.index.month.map(seasonal_indexes)
decomp.head(10)<jupyter_output><empty_output><jupyter_text>**Step 4: Calculate the remainder**The irregular component $E_t$ is calculated by subtracting the trend and seasonal components from the actual.$E_t = Y_t - T_t - S_t$<jupyter_code>decomp['resid'] = decomp['actual'] - decomp['trend'] - decomp['seasonal']<jupyter_output><empty_output><jupyter_text>Seasonal Decomposition Plot<jupyter_code>fig, axes = plt.subplots(4, 1, sharex=True, figsize=(12,9))
axes[0].plot(decomp['actual'])
axes[1].plot(decomp['trend'])
axes[2].plot(decomp['seasonal'])
axes[3].plot(decomp['resid'])

axes[0].set_ylabel('actual')
axes[1].set_ylabel('trend-cycle')
axes[2].set_ylabel('seasonal')
axes[3].set_ylabel('remainder')<jupyter_output><empty_output><jupyter_text>Seasonal Adjustment: A side product of decomposition$Y_t - S_t = T_t + E_t$Many economic time series are publishing using a seasonally adjusted data.  This is achieved by subtrack the seasonal indicies from the observerd time series.<jupyter_code>seasonal_adj = decomp['actual'] - decomp['seasonal']
fig, axes = plt.subplots(2, 1, sharex=True, figsize=(12,8))
axes[0].plot(arrival_rate)
axes[1].plot(seasonal_adj)<jupyter_output><empty_output><jupyter_text>Multiplicative Seasonality<jupyter_code>sales = pd.read_csv('data/Alcohol_Sales.csv', index_col='DATE', parse_dates=True)
sales.index.freq = 'MS'
sales_rate = sales['sales'] / sales.index.days_in_month
sales_rate.plot(figsize=(12,4))<jupyter_output><empty_output><jupyter_text>**Step 1: Estimate the trend component**As before we will do this using a 12 month 'centered' moving average.<jupyter_code>decomp = pd.DataFrame(sales_rate)
decomp.columns = ['actual']
decomp['trend'] = sales_rate.rolling(window=12, center=True).mean()
ax = decomp.trend.plot(figsize=(12,4))
sales_rate.plot(ax=ax)<jupyter_output><empty_output><jupyter_text>**Step 2: Detrend the series**We first need to **de-trend** the time series.  In the additive model we do this by **dividing** the trend from the actual observations of the time series.$Y_t = T_t \cdot  S_t \cdot  E_t$<jupyter_code>detrended = decomp['actual'] / decomp['trend']
detrended.plot(figsize=(12,4))<jupyter_output><empty_output><jupyter_text>**Step 3: Calculate the seasonal indicies**We again work with the detrended series. To calculate the seasonal indicies we group by month and take the average of the detrended series. Note that the indicies are now proportion above and below the mean level of the series.<jupyter_code>seasonal_indexes = detrended.groupby(by=detrended.index.month).mean()
seasonal_indexes.plot(figsize=(12,4))<jupyter_output><empty_output><jupyter_text>We then want to **map** the seasonal indicies to the rows in the time series.<jupyter_code>decomp['seasonal'] = decomp.index.month.map(seasonal_indexes)
decomp.head(10)<jupyter_output><empty_output><jupyter_text>**Step 4: Calculate the remainder**The irregular component $E_t$ is calculated by dividing by the trend and seasonal components from the actual.$E_t = \frac{Y_t}{T_t S_t}$<jupyter_code>decomp['resid'] = decomp['actual'] / (decomp['trend'] * decomp['seasonal'])
decomp.head()<jupyter_output><empty_output><jupyter_text>Seasonal decomp plot<jupyter_code>fig, axes = plt.subplots(4, 1, sharex=True, figsize=(12,9))
axes[0].plot(decomp['actual'])
axes[1].plot(decomp['trend'])
axes[2].plot(decomp['seasonal'])
axes[3].plot(decomp['resid'])

axes[0].set_ylabel('actual')
axes[1].set_ylabel('trend-cycle')
axes[2].set_ylabel('seasonal')
axes[3].set_ylabel('remainder')<jupyter_output><empty_output><jupyter_text>Seasonal Adjustment.$\frac{Y_t}{S_t} = T_t \cdot E_t$Many economic time series are publishing using a seasonally adjusted data.  This is achieved by dividing the observed time series by the seasonal indicies.<jupyter_code>seasonal_adj = decomp['actual'] / decomp['seasonal']
seasonal_adj.plot(figsize=(12,4))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1087495110183857, 'fraction_numerical': 0.011344373451558221, 'mean_word_length': 7.05672268907563, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 78, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '665174', 'license': 'MIT', 'n_tokens_mistral': 2584, 'n_tokens_neox': 2430, 'n_words': 732, 'path': '01_basics/01_additional_lecture_notes/Bonus_ClassicalSeasonalDecomposition.ipynb', 'repo_name': 'health-data-science-OR/forecasting', 'size': '7669'}