<jupyter_start><jupyter_text>Time Series Decomposition<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline

df = pd.read_csv('ConstructionTimeSeriesDatav2.csv')<jupyter_output><empty_output><jupyter_text>Public Construction Spending - Basic Decomposition Method First, take a look at the data using a simple visualization.<jupyter_code>#View the data

plt.plot(df['Month'],df['Public Construction'])<jupyter_output><empty_output><jupyter_text>The graph shows the data has a linear upward trend with sesaonal cycles.  Now generate the intercept and slope values.<jupyter_code>from scipy import stats

slope, intercept, r_value, p_value, std_err = stats.linregress(df.index,df['Public Construction'])
print('intercept =', intercept, '    slope =', slope, '     p_value = ',p_value)<jupyter_output>intercept = 18327.939893765724     slope = 52.00857586303297      p_value =  3.028349007007645e-09<jupyter_text>Create a new column that calculates a simple regression.<jupyter_code>#Create the R1 regression column
def create_regress_col(row, intercept, slope):
    return float(intercept) + float(row['Month']) * slope
    
df['regress'] = df.apply(create_regress_col,args = (intercept,slope),axis = "columns")
df['R1'] = df['Public Construction'] - df['regress']
df.style.format({
    'Public Construction': '{:,.0f}'.format,
    'regress': '{:,.0f}'.format,
    'R1': '{:,.0f}'.format
})<jupyter_output><empty_output><jupyter_text>Plot the results of the R1 simple regression column.<jupyter_code>plt.plot(df.index,df.R1)<jupyter_output><empty_output><jupyter_text>The resluts of R1 show a seasonality pattern that repeats about every 12 months. Check the seasonality using auto-correlation.<jupyter_code># Create column with lag of 12
lag = 12
df['lag12'] = np.NaN
for i in range(len(df['lag12']))[lag:]:
    df['lag12'].iloc[i] = df['Public Construction'].iloc[i-4]
print(df.head(n=10))

# Compute autocorrelations
for i in range(int(len(df.index)/2)):
    print('autocorrelation, lag =',i,':',df.R1.autocorr(lag = i))
    
fig,ax = plt.subplots()
ax.plot(df['Month'],df['Public Construction'],c='k')
ax.plot(df['Month'],df.lag12,c='b')
ax.set_xlim([0,150])
ax.text(155,17000,'Public Construction',color='k')
ax.text(155,26000,'Lagged\nPublic Construction',color='b')
ax.set_xlabel('Month')<jupyter_output>C:\Users\rober\Anaconda3\lib\site-packages\pandas\core\indexing.py:189: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self._setitem_with_indexer(indexer, value)<jupyter_text>The lagged public construction is highly correlated to the actual Public Construction spending amounts.Plot each subset of the anual pattern.<jupyter_code>dfQtr = pd.DataFrame()
cycleLen = 12
for i in range(int(len(df.index)/cycleLen)):
    newData = pd.DataFrame({i:df['R1'].iloc[i*cycleLen:(i+1)*cycleLen]})
    newData.index = range(0,len(newData))
    dfQtr = pd.concat([dfQtr,newData],axis=1)

fig,ax = plt.subplots()
ax.plot(dfQtr)<jupyter_output><empty_output><jupyter_text>This confirms that the 12 seasonal patterns look the same.Next, average the spending for each of the years in the data.<jupyter_code>avg = []
for i in range(len(dfQtr.index)):
    avg.append(dfQtr.iloc[i].mean())

dfQtr = pd.concat([dfQtr,pd.DataFrame({'avg':avg})], axis=1)
print(dfQtr)

fig,ax = plt.subplots()
c = 180
for col in dfQtr.columns.values:
    if col == 'avg':
        ax.plot(dfQtr[col], c = 'r')
    else:
        ax.plot(dfQtr[col], c = 'k')<jupyter_output>0            1            2            3            4  \
0  -4136.948470 -4600.051380 -5669.154290 -5872.257201 -5157.360111   
1  -4318.957045 -5150.059956 -5991.162866 -5787.265777 -5071.368687   
2  -4097.965621 -4598.068532 -4289.171442 -4436.274352 -3288.377263   
3  -2314.974197 -2716.077108 -2607.180018 -2752.282928 -1488.385839   
4   -212.982773 -1025.085683 -1127.188594  -363.291504   922.605585   
5    741.008651  1036.905741  1004.802830  1862.699920  3274.597010   
6   1677.000075  1438.897165  1503.794254  2114.691344  3534.588434   
7   2762.991499  2588.888589  2171.785679  3461.682768  4544.579858   
8   2588.982923  2654.880013  1831.777103  3016.674192  3630.571282   
9    838.974348   848.871437  -370.231473   798.665617  2039.562706   
10 -1090.034228 -1723.137139 -1339.240049  -997.342959   481.554130   
11 -2991.042804 -4110.145714 -3905.248625 -3345.351535 -1984.454446   

              5            6            7            8            9  \
0  -3[...]<jupyter_text>Now, calculate the seasonal affect for each year of spending.<jupyter_code>df['S'] = np.NaN
df['R2Public'] = np.NaN
df['Composite'] = np.NaN
df['R2ErrorPublic'] = np.NaN
S = dfQtr['avg'].tolist()
for i in df.index:
    df.loc[i,'S'] = S[i%cycleLen]
    df.loc[i,'R2X'] = df.loc[i,'R1'] - df.loc[i,'S']
    df.loc[i,'Composite'] = df.loc[i,'regress'] + df.loc[i,'S']
    df.loc[i,'R2ErrorX'] = 100*df.loc[i,'R2X'] / df.loc[i,'Public Construction']
df.style.format({
    'Public Construction': '{:,.0f}'.format,
    'regress': '{:,.0f}'.format,
    'R1': '{:,.0f}'.format,
    'S': '{:,.0f}'.format,
    'R2Public': '{:,.0f}'.format,
    'Composite':'{:,.0f}'.format,
    'R2ErrorPublic': '{:.2f}%'.format
})<jupyter_output><empty_output><jupyter_text>Visualize how the model fits the original Public Construction spending series.<jupyter_code>fig, ax = plt.subplots()
ax.plot(df['Public Construction'],c='k')
ax.plot(df['Composite'],c='b')
ax.set_xlim([0,150])
ax.text(150,16000,'Public Construction', color='k')
ax.text(150,20500,'Model', color='b')
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.set_xlabel('Months')
ax.set_ylabel('Construction Spending (in millions of $)')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15688602442333785, 'fraction_numerical': 0.13076662143826323, 'mean_word_length': 4.4400369003690034, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '512171', 'license': 'CC0-1.0', 'n_tokens_mistral': 2708, 'n_tokens_neox': 2269, 'n_words': 451, 'path': 'timeseriesdecom/Public_TimeSeriesDecomposition.ipynb', 'repo_name': 'robertbell33wp/robertbell33wp.github.io', 'size': '5896'}