<jupyter_start><jupyter_text>_Lambda School Data Science — Tree Ensembles_  Decision Trees Assignment  Part 1: House Price RegressionApply decision trees to the Ames housing dataset you've worked with this week!- Try multiple features- Try features you've engineered- Try different `max_depth` paramaters- What's the best Test Root Mean Squared Error you can get? *Share with your cohort on Slack!*- What's a cool visualization you can make? *Share with your cohort on Slack!*<jupyter_code>!pip install graphviz
!apt-get install graphviz
# %matplotlib inline
import graphviz
from IPython.display import display
from ipywidgets import interact
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor, export_graphviz
from sklearn.metrics import mean_squared_error
# set dataframe display options
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)

# import data
url = 'https://raw.githubusercontent.com/ryanleeallred/datasets/master/Ames%20Housing%20Data/train.csv'
df = pd.read_csv(url)
# Remove categorical data
df = df.select_dtypes(include='number')
# Remove columns with NaN values
df = df.drop(columns=['LotFrontage', 'MasVnrArea', 'GarageYrBlt', 'Id'])
# View columns
df.columns
# Separate X and y variables
target = 'SalePrice'
features = ['MSSubClass', 'LotArea', 'OverallQual', 'OverallCond', 'YearBuilt',
       'YearRemodAdd', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF',
       '1stFlrSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath',
       'BsmtHalfBath', 'FullBath', 'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr',
       'TotRmsAbvGrd', 'Fireplaces', 'GarageCars', 'GarageArea', 'WoodDeckSF',
       'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea',
       'MiscVal', 'MoSold', 'YrSold']

X = df[feature]
y = df[target]<jupyter_output><empty_output><jupyter_text>Baseline Decision Tree<jupyter_code># Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

# Fit the model
tree = DecisionTreeRegressor(max_depth=2)
tree.fit(X_train, y_train)

# Predict y with X_test
y_pred = tree.predict(X_test)

# Print RMSE and R^2 scores for train and test data
print('Train R^2 Score:', tree.score(X_train, y_train))
print('Test R^2 Score:', tree.score(X_test, y_test))
print('Root Mean Squared Error:', np.sqrt(mean_squared_error(y_test, y_pred)))<jupyter_output>Train R^2 Score: 0.6217396608043659
Test R^2 Score: 0.6613143997312394
Root Mean Squared Error: 48709.3143193345<jupyter_text>Decision Tree to Max Test R^2 Score<jupyter_code># Make a copy of the dataframe
df_copy = df.copy()

# Engineered Features
df_copy['age'] = 2010 - df['YearBuilt']
df_copy['bedroom_bathroom_ratio'] = df['BedroomAbvGr'] / (df['FullBath'] + df['HalfBath'] + df['BsmtFullBath'] + df['BsmtHalfBath'])
df_copy['bedroom_per_sqft'] = df['BedroomAbvGr'] / (df['1stFlrSF'] + df['2ndFlrSF'] + df['BsmtFinSF1'] + df['BsmtFinSF2'])
df_copy['bathrooms_per_sqft'] = (df['BsmtFullBath'] + df['BsmtHalfBath'] + df['FullBath'] + df['HalfBath']) / (df['1stFlrSF'] + df['2ndFlrSF'] + df['TotalBsmtSF'])

# Polynomial Features
df_copy['overall_qual_squared'] = df['OverallQual'] ** 2
df_copy['overall_cond_squared'] = df['OverallCond'] ** 2
df_copy['GrLivArea_squared'] = df['GrLivArea'] ** 2
df_copy['first_floor_sqft_squared'] = df['1stFlrSF'] ** 2
df_copy['second_floor_sqft_squared'] = df['2ndFlrSF'] ** 2
# View columns
df_copy.columns
# Convert price to natural log
df_copy['ln_price'] = np.log(df_copy['SalePrice'])

# Separate X varibles and y variable
target = 'ln_price'
features = ['MSSubClass', 'OverallQual', 'OverallCond', 'BsmtFinSF1', 
       'BsmtFinSF2','BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF', 
       'GrLivArea', 'BsmtFullBath','FullBath',  'KitchenAbvGr', 'TotRmsAbvGrd', 
       'Fireplaces', 'GarageCars', 'GarageArea', 'WoodDeckSF','OpenPorchSF', 
       'EnclosedPorch', 'ScreenPorch', 'PoolArea', 'MoSold', 'YrSold', 'age',  
       'bedroom_bathroom_ratio', 'bathrooms_per_sqft',  'overall_qual_squared',
       'overall_cond_squared', 'GrLivArea_squared',  'first_floor_sqft_squared']

X = df_copy[features]
y = df_copy[target]

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

# Fit the model
tree = DecisionTreeRegressor(max_depth=6, random_state=42)
tree.fit(X_train, y_train)

# Predict y with X_test
y_pred = tree.predict(X_test)

# Print R^2 Scores
print('Train R^2 Score:', tree.score(X_train, y_train))
print('Test R^2 Score:', tree.score(X_test, y_test))
print('Root Mean Squared Error:', np.sqrt(mean_squared_error(y_test, y_pred)))<jupyter_output>Train R^2 Score: 0.8885280975650572
Test R^2 Score: 0.8069334504146474
Root Mean Squared Error: 0.18284225302075302<jupyter_text>Visualization<jupyter_code>def viz3D(fitted_model, df, feature1, feature2, target='', num=100):
    """
    Visualize model predictions in 3D, for regression or binary classification
    
    Parameters
    ----------
    fitted_model : scikit-learn model, already fitted
    df : pandas dataframe, which was used to fit model
    feature1 : string, name of feature 1
    feature2 : string, name of feature 2
    target : string, name of target
    num : int, number of grid points for each feature
    
    References
    ----------
    https://jakevdp.github.io/PythonDataScienceHandbook/04.12-three-dimensional-plotting.html
    https://scikit-learn.org/stable/auto_examples/tree/plot_iris.html  
    """
    x1 = np.linspace(df[feature1].min(), df[feature1].max(), num)
    x2 = np.linspace(df[feature2].min(), df[feature2].max(), num)
    X1, X2 = np.meshgrid(x1, x2)
    X = np.c_[X1.flatten(), X2.flatten()]
    if hasattr(fitted_model, 'predict_proba'):
        predicted = fitted_model.predict_proba(X)[:,1]
    else:
        predicted = fitted_model.predict(X)
    Z = predicted.reshape(num, num)
    
    fig = plt.figure()
    ax = plt.axes(projection='3d')
    ax.plot_surface(X1, X2, Z, cmap='viridis')
    ax.set_xlabel(feature1)
    ax.set_ylabel(feature2)
    ax.set_zlabel(target)
    return fig
target = 'SalePrice'
features = ['OverallQual', 'bedroom_bathroom_ratio']

X = df_copy[features]
y = df_copy[target]
# Fit the model
tree = DecisionTreeRegressor(max_depth=2, random_state=42)
tree.fit(X, y)

# Plot the tree
import matplotlib.pyplot as plt
viz3D(tree, df_copy, feature1='OverallQual', feature2='bedroom_bathroom_ratio', target='ln_price')<jupyter_output><empty_output><jupyter_text>Part 2 / Stretch: "Play Tennis" Classification We'll reproduce the "Play Tennis" example from Ross Quinlan's 1986 paper, [Induction of Decison Trees](https://link.springer.com/content/pdf/10.1007%2FBF00116251.pdf).[According to Wikipedia](https://en.wikipedia.org/wiki/Ross_Quinlan), "<NAME> is a computer science researcher in data mining and decision theory. He has contributed extensively to the development of decision tree algorithms, including inventing the canonical C4.5 and ID3 algorithms."  "Table 1 shows a small training set"<jupyter_code>import pandas as pd

columns = 'No. Outlook Temperature Humidity Windy PlayTennis'.split()

raw = """1 sunny hot high false N
2 sunny hot high true N
3 overcast hot high false P
4 rain mild high false P
5 rain cool normal false P
6 rain cool normal true N
7 overcast cool normal true P
8 sunny mild high false N
9 sunny cool normal false P
10 rain mild normal false P
11 sunny mild normal true P
12 overcast mild high true P
13 overcast hot normal false P
14 rain mild high true N"""

data = [row.split() for row in raw.split('\n')]
tennis = pd.DataFrame(data=data, columns=columns).set_index('No.')
tennis['PlayTennis'] = (tennis['PlayTennis'] == 'P').astype(int)

tennis<jupyter_output><empty_output><jupyter_text>"A decision tree that correctly classifies each object in the training set is given in Figure 2." In this dataset, the tennis player decided to play on 64% of the days, and decided not to on 36% of the days.<jupyter_code>tennis['PlayTennis'].value_counts(normalize=True) * 100<jupyter_output><empty_output><jupyter_text>The tennis player played on 100% of the overcast days, 40% of the sunny days, and 60% of the  rainy days<jupyter_code>tennis.groupby('Outlook')['PlayTennis'].mean() * 100<jupyter_output><empty_output><jupyter_text>On sunny days, the tennis player's decision depends on the humidity. (The Outlook and Humidity features interact.)<jupyter_code>sunny = tennis[tennis['Outlook']=='sunny']
sunny.groupby('Humidity')['PlayTennis'].mean() * 100<jupyter_output><empty_output><jupyter_text>On rainy days, the tennis player's decision depends on the wind. (The Outlook and Windy features interact.)<jupyter_code>rainy = tennis[tennis['Outlook']=='rain']
rainy.groupby('Windy')['PlayTennis'].mean() * 100<jupyter_output><empty_output><jupyter_text>Before modeling, we will ["encode" categorical variables, using pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/reshaping.html?highlight=get_dummiescomputing-indicator-dummy-variables)<jupyter_code>y = tennis['PlayTennis']
X = pd.get_dummies(tennis.drop(columns='PlayTennis'))
X<jupyter_output><empty_output><jupyter_text>Train a Decision Tree ClassifierGet a score of 100% (accuracy)https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html<jupyter_code><jupyter_output><empty_output><jupyter_text>Compare to Logistic Regressionhttps://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html<jupyter_code><jupyter_output><empty_output><jupyter_text>Visualize the treehttps://scikit-learn.org/stable/modules/generated/sklearn.tree.export_graphviz.html<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12529952076677317, 'fraction_numerical': 0.027356230031948883, 'mean_word_length': 5.961084086170952, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '162830', 'license': 'MIT', 'n_tokens_mistral': 3789, 'n_tokens_neox': 3497, 'n_words': 856, 'path': 'module4-decision-trees/LS_DS_411A_Decision_Trees.ipynb', 'repo_name': 'JLDaniel77/DS-Unit-2-Sprint-2-Regression', 'size': '10027'}