<jupyter_start><jupyter_text>Exercise 30 - Dummy VariablesJust because we are solving a regression problem with continuous data it does not mean we cant integrate some categorical variables into our model.  These particular categorical variables, known as *Dummy Variables* are constructed from the continuous values in our dataset.  In this example we are going to create two categorical variables, one indicating if the year is greater than 1960 and one if the year is greater than 1940.  These two years were chosen as limits for creating the dummy variables because of the shape of the 15 year moving average.  As discussed there seems to be the start of an increasing trend around 1960 and 1940 was sellected as this is the approximate end of the early plateau.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression<jupyter_output><empty_output><jupyter_text>Reload the data from the previous exercise<jupyter_code>df = pd.read_csv('synth_temp.csv')
df = df.loc[df.Year > 1901]
df_group_year = df.groupby('Year').agg(np.mean)

# Moving average
window = 10
rolling = df_group_year.AverageTemperature.rolling(window).mean();

df_group_year['Year'] = df_group_year.index

model = LinearRegression()
df_group_year['Gt_1960'] = [0 if year < 1960 else 10 for year in df_group_year.Year] # Dummy Variable - greater than 1960
df_group_year.head(n=2)
df_group_year['Gt_1945'] = [0 if year < 1945 else 10 for year in df_group_year.Year]# Dummy Variable - greater than 1945
df_group_year.head(n=2)
df_group_year.tail(n=2)
# Note the year values need to be provided as an N x 1 array
model.fit(df_group_year[['Year', 'Gt_1960', 'Gt_1945']], df_group_year.AverageTemperature)
# Note the year values need to be provided as an N x 1 array
r2 = model.score(df_group_year[['Year', 'Gt_1960', 'Gt_1945']], df_group_year.AverageTemperature)
print(f'r2 score = {r2:0.4f}')
# Use linspace to get a range of values, in 20 yr increments
x = np.linspace(df_group_year['Year'].min(), df_group_year['Year'].max(), 20)
x
trend_x = np.zeros((20, 3))
trend_x[:,0] = x # Assign to the first column
trend_x[:,1] = [10 if _x > 1960 else 0 for _x in x] # Assign to the second column
trend_x[:,2] = [10 if _x > 1945 else 0 for _x in x] # Assign to the third column 
trend_x
trend_y = model.predict(trend_x)
trend_y
fig = plt.figure(figsize=(10, 7))
ax = fig.add_axes([1, 1, 1, 1]);

# Temp measurements
ax.scatter(df_group_year.index, df_group_year.AverageTemperature, label='Raw Data', c='k');
ax.plot(df_group_year.index, rolling, c='k', linestyle='--', label=f'{window} year moving average');
ax.plot(trend_x[:,0], trend_y, c='k', label='Model: Predicted trendline')

ax.set_title('Mean Air Temperature Measurements')
ax.set_xlabel('Year')
ax.set_ylabel('Temperature (degC)')
ax.set_xticks(range(df_group_year.index.min(), df_group_year.index.max(), 10))
ax.legend();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10806233062330624, 'fraction_numerical': 0.03929539295392954, 'mean_word_length': 5.711363636363636, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '304359', 'license': 'MIT', 'n_tokens_mistral': 1086, 'n_tokens_neox': 976, 'n_words': 338, 'path': 'Chapter 3 - Regression Analysis/Exercise 30 - Dummy Variables.ipynb', 'repo_name': 'doc-E-brown/Applied-Supervised-Learning-with-Python', 'size': '2952'}