<jupyter_start><jupyter_text>This lab on Decision Trees is a Python adaptation of p. 324-331 of "Introduction to Statistical Learning withApplications in R" by <NAME>, <NAME>, <NAME> and <NAME>. Original adaptation by <NAME>, updated by <NAME> at SmithCollege for SDS293: Machine Learning (Spring 2016).<jupyter_code>import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import graphviz

%matplotlib inline<jupyter_output><empty_output><jupyter_text>8.3.1 Fitting Classification TreesThe ${\tt sklearn}$ library has a lot of useful tools for constructing classification and regression trees:<jupyter_code>from sklearn.cross_validation import train_test_split
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.metrics import confusion_matrix, mean_squared_error<jupyter_output><empty_output><jupyter_text>We'll start by using **classification trees** to analyze the ${\tt Carseats}$ data set. In thesedata, ${\tt Sales}$ is a continuous variable, and so we begin by converting it to abinary variable. We use the ${\tt ifelse()}$ function to create a variable, called${\tt High}$, which takes on a value of ${\tt Yes}$ if the ${\tt Sales}$ variable exceeds 8, andtakes on a value of ${\tt No}$ otherwise. We'll append this onto our dataFrame using the ${\tt .map()}$ function, and then do a little data cleaning to tidy things up:<jupyter_code>df3 = pd.read_csv('Carseats.csv').drop('Unnamed: 0', axis=1)
df3['High'] = df3.Sales.map(lambda x: 1 if x>8 else 0)
df3.ShelveLoc = pd.factorize(df3.ShelveLoc)[0]
df3.Urban = df3.Urban.map({'No':0, 'Yes':1})
df3.US = df3.US.map({'No':0, 'Yes':1})
df3.info()<jupyter_output><empty_output><jupyter_text>In order to properly evaluate the performance of a classification tree onthe data, we must estimate the test error rather than simply computingthe training error. We first split the observations into a training set and a testset:<jupyter_code>X = df3.drop(['Sales', 'High'], axis=1)
y = df3.High

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, random_state=0)<jupyter_output><empty_output><jupyter_text>We now use the ${\tt DecisionTreeClassifier()}$ function to fit a classification tree in order to predict${\tt High}$ using all variables but ${\tt Sales}$ (that would be a little silly...). Unfortunately, manual pruning is not implemented in ${\tt sklearn}$: http://scikit-learn.org/stable/modules/tree.htmlHowever, we can limit the depth of a tree using the ${\tt max\_depth}$ parameter:<jupyter_code>clf = DecisionTreeClassifier(max_depth=6)
clf.fit(X_train, y_train)
clf.score(X_train, y_train)<jupyter_output><empty_output><jupyter_text>We see that the training accuracy is 95.5%.One of the most attractive properties of trees is that they can begraphically displayed. Unfortunately, this is a bit of a roundabout process in ${\tt sklearn}$. We use the ${\tt export\_graphviz()}$ function to export the tree structure to a temporary ${\tt .dot}$ file,and the ${\tt graphviz.Source()}$ function to display the image:<jupyter_code>export_graphviz(clf, out_file="mytree.dot", feature_names=X_train.columns)
with open("mytree.dot") as f:
    dot_graph = f.read()
graphviz.Source(dot_graph)<jupyter_output><empty_output><jupyter_text>The most important indicator of ${\tt High}$ sales appears to be ${\tt Price}$. Finally, let's evaluate the tree's performance onthe test data. The ${\tt predict()}$ function can be used for this purpose. We can then build a confusion matrix, which shows that we are making correct predictions foraround 74.5% of the test data set:<jupyter_code>pred = clf.predict(X_test)
cm = pd.DataFrame(confusion_matrix(y_test, pred).T, index=['No', 'Yes'], columns=['No', 'Yes'])
print(cm)
# 99+50/200 = 0.745<jupyter_output><empty_output><jupyter_text>8.3.2 Fitting Regression TreesNow let's try fitting a **regression tree** to the ${\tt Boston}$ data set from the ${\tt MASS}$ library. First, we create atraining set, and fit the tree to the training data using ${\tt medv}$ (median home value) as our response:<jupyter_code>boston_df = pd.read_csv('Boston.csv')
X = boston_df.drop('medv', axis=1)
y = boston_df.medv
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, random_state=0)

# Pruning not supported. Choosing max depth 2)
regr2 = DecisionTreeRegressor(max_depth=2)
regr2.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Let's take a look at the tree:<jupyter_code>export_graphviz(regr2, out_file="mytree.dot", feature_names=X_train.columns)
with open("mytree.dot") as f:
    dot_graph = f.read()
graphviz.Source(dot_graph)<jupyter_output><empty_output><jupyter_text>The variable ${\tt lstat}$ measures the percentage of individuals with lowersocioeconomic status. The tree indicates that lower values of ${\tt lstat}$ correspondto more expensive houses. The tree predicts a median house priceof \$45,766 for larger homes (${\tt rm}>=7.435$) in suburbs in which residents have high socioeconomicstatus (${\tt lstat}<7.81$).Now let's see how it does on the test data:<jupyter_code>pred = regr2.predict(X_test)

plt.scatter(pred, y_test, label='medv')
plt.plot([0, 1], [0, 1], '--k', transform=plt.gca().transAxes)
plt.xlabel('pred')
plt.ylabel('y_test')

mean_squared_error(y_test, pred)<jupyter_output><empty_output><jupyter_text>The test set MSE associated with the regression tree is28.8. The square root of the MSE is therefore around 5.37, indicatingthat this model leads to test predictions that are within around \$5,370 ofthe true median home value for the suburb.     8.3.3 Bagging and Random ForestsLet's see if we can improve on this result using **bagging** and **random forests**. The exact results obtained in this section maydepend on the version of ${\tt python}$ and the version of the ${\tt RandomForestRegressor}$ packageinstalled on your computer, so don't stress out if you don't match up exactly with the book. Recall that **bagging** is simply a special case ofa **random forest** with $m = p$. Therefore, the ${\tt RandomForestRegressor()}$ function canbe used to perform both random forests and bagging. Let's start with bagging:<jupyter_code># Bagging: using all features
regr1 = RandomForestRegressor(max_features=13, random_state=1)
regr1.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>The argument ${\tt max\_features=13}$ indicates that all 13 predictors should be consideredfor each split of the tree -- in other words, that bagging should be done. Howwell does this bagged model perform on the test set?<jupyter_code>pred = regr1.predict(X_test)
plt.scatter(pred, y_test, label='medv')
plt.plot([0, 1], [0, 1], '--k', transform=plt.gca().transAxes)
plt.xlabel('pred')
plt.ylabel('y_test')
mean_squared_error(y_test, pred)<jupyter_output><empty_output><jupyter_text>The test setMSE associated with the bagged regression tree is significantly lower than our single tree! We can grow a random forest in exactly the same way, except thatwe'll use a smaller value of the ${\tt max\_features}$ argument. Here we'lluse ${\tt max\_features = 6}$:<jupyter_code># Random forests: using 6 features
regr2 = RandomForestRegressor(max_features=6, random_state=1)
regr2.fit(X_train, y_train)

pred = regr2.predict(X_test)
mean_squared_error(y_test, pred)<jupyter_output><empty_output><jupyter_text>The test set MSE is even lower; this indicates that random forests yielded animprovement over bagging in this case.Using the ${\tt feature\_importances\_}$ attribute of the ${\tt RandomForestRegressor}$, we can view the importance of eachvariable:<jupyter_code>Importance = pd.DataFrame({'Importance':regr2.feature_importances_*100}, index=X.columns)
Importance.sort_values(by='Importance', axis=0, ascending=True).plot(kind='barh', color='r', )
plt.xlabel('Variable Importance')
plt.gca().legend_ = None<jupyter_output><empty_output><jupyter_text>The results indicate that across all of the trees considered in the randomforest, the wealth level of the community (${\tt lstat}$) and the house size (${\tt rm}$)are by far the two most important variables.  8.3.4 BoostingNow we'll use the ${\tt GradientBoostingRegressor}$ package to fit **boostedregression trees** to the ${\tt Boston}$ data set. Theargument ${\tt n_estimators=500}$ indicates that we want 500 trees, and the option${\tt interaction.depth=4}$ limits the depth of each tree:<jupyter_code>regr = GradientBoostingRegressor(n_estimators=500, learning_rate=0.01, max_depth=4, random_state=1)
regr.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Let's check out the feature importances again:<jupyter_code>feature_importance = regr.feature_importances_*100
rel_imp = pd.Series(feature_importance, index=X.columns).sort_values(inplace=False)
rel_imp.T.plot(kind='barh', color='r', )
plt.xlabel('Variable Importance')
plt.gca().legend_ = None<jupyter_output><empty_output><jupyter_text>We see that ${\tt lstat}$ and ${\tt rm}$ are again the most important variables by far. Now let's use the boosted model to predict ${\tt medv}$ on the test set:<jupyter_code>mean_squared_error(y_test, regr.predict(X_test))<jupyter_output><empty_output><jupyter_text>The test MSE obtained is similar to the test MSE for random forestsand superior to that for bagging. If we want to, we can perform boostingwith a different value of the shrinkage parameter $\lambda$. Here we take $\lambda = 0.2$:<jupyter_code>regr2 = GradientBoostingRegressor(n_estimators=500, learning_rate=0.2, max_depth=4, random_state=1)
regr2.fit(X_train, y_train)
mean_squared_error(y_test, regr2.predict(X_test))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1118421052631579, 'fraction_numerical': 0.01665296052631579, 'mean_word_length': 6.415396341463414, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 81, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '597419', 'license': 'MIT', 'n_tokens_mistral': 3228, 'n_tokens_neox': 3047, 'n_words': 1042, 'path': 'Lab 14 - Decision Trees in Python.ipynb', 'repo_name': 'DevOM3/islr-python', 'size': '9779'}