<jupyter_start><jupyter_text>Decision Tree - Regression---------------------------------------------------------------------you can Find me on Github:>  [ GitHub](https://github.com/lev1khachatryan)      Notebook  Content- [8.1.1 Regression Trees](8.1.1-Regression-Trees)- [8.1.2 Classification Trees](8.1.2-Classification-Trees)- [Lab: 8.3.1 Fitting Classification Trees](8.3.1-Fitting-Classification-Trees)- [Lab: 8.3.2 Fitting Regression Trees](8.3.2-Fitting-Regression-Trees)- [Lab: 8.3.3 Bagging and Random Forests](8.3.3-Bagging-and-Random-Forests)- [Lab: 8.3.4 Boosting](8.3.4-Boosting)     Decision tree builds regression or classification models in the form of a tree structure. It breaks down a dataset into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed. The final result is a tree with decision nodes and leaf nodes. A decision node (e.g., Outlook) has two or more branches (e.g., Sunny, Overcast and Rainy), each representing values for the attribute tested. Leaf node (e.g., Hours Played) represents a decision on the numerical target. The topmost decision node in a tree which corresponds to the best predictor called root node. Decision trees can handle both categorical and numerical data.        Decision Tree Algorithm   The core algorithm for building decision trees called ***ID3*** by <NAME> which employs a top-down, greedy search through the space of possible branches with no backtracking. The ***ID3 algorithm can be used to construct a decision tree for regression by replacing Information Gain with Standard Deviation Reduction***.  Standard Deviation   A decision tree is built top-down from a root node and involves partitioning the data into subsets that contain instances with similar values (homogenous). We use standard deviation to calculate the homogeneity of a numerical sample. ***If the numerical sample is completely homogeneous its standard deviation is zero.***    a) Standard deviation for one attribute:      * Standard Deviation (S) is for tree building (branching).* Coefficient of Deviation (CV) is used to decide when to stop branching. We can use Count (n) as well.* Average (Avg) is the value in the leaf nodes.      b) Standard deviation for two attributes (target and predictor):         Standard Deviation Reduction   The standard deviation reduction is based on the decrease in standard deviation after a dataset is split on an attribute. Constructing a decision tree is all about finding attribute that returns the highest standard deviation reduction (i.e., the most homogeneous branches).    Step 1: The standard deviation of the target is calculated.  Standard deviation (Hours Played) = 9.32    Step 2: The dataset is then split on the different attributes. The standard deviation for each branch is calculated. The resulting standard deviation is subtracted from the standard deviation before the split. The result is the standard deviation reduction.         Step 3: The attribute with the largest standard deviation reduction is chosen for the decision node.          Step 4a: The dataset is divided based on the values of the selected attribute. This process is run recursively on the non-leaf branches, until all data is processed.        In practice, we need some termination criteria. For example, when coefficient of deviation (CV) for a branch becomes smaller than a certain threshold (e.g., 10%) and/or when too few instances (n) remain in the branch (e.g., 3).     Step 4b: "Overcast" subset does not need any further splitting because its CV (8%) is less than the threshold (10%). The related leaf node gets the average of the "Overcast" subset.         Step 4c: However, the "Sunny" branch has an CV (28%) more than the threshold (10%) which needs further splitting. We select "Windy" as the best best node after "Outlook" because it has the largest SDR.         Because the number of data points for both branches (FALSE and TRUE) is equal or less than 3 we stop further branching and assign the average of each branch to the related leaf node.         Step 4d: Moreover, the "rainy" branch has an CV (22%) which is more than the threshold (10%). This branch needs further splitting. We select "Windy" as the best best node because it has the largest SDR.         Because the number of data points for all three branches (Cool, Hot and Mild) is equal or less than 3 we stop further branching and assign the average of each branch to the related leaf node.      When the number of instances is more than one at a leaf node we calculate the average as the final value for the target.<jupyter_code># %load ../standard_import.txt
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

import pydot
import pydotplus
from IPython.display import Image

from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.externals.six import StringIO  
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier, BaggingRegressor, RandomForestRegressor, GradientBoostingRegressor
from sklearn.metrics import mean_squared_error,confusion_matrix, classification_report

%matplotlib inline
plt.style.use('seaborn-white')
# This function creates images of tree models using pydot
def print_tree(estimator, features, class_names=None, filled=True):
    tree = estimator
    names = features
    color = filled
    classn = class_names
    
    dot_data = StringIO()
    export_graphviz(estimator, out_file=dot_data, feature_names=features, class_names=classn, filled=filled)
#     graph = pydot.graph_from_dot_data(dot_data.getvalue())
    graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
    return(graph)<jupyter_output><empty_output><jupyter_text>8.1.1 Regression Trees<jupyter_code>df = pd.read_csv('input/20190903/ISLR_Datasets/Hitters.csv').dropna()
df.info()
X = df[['Years', 'Hits']].as_matrix()
y = np.log(df.Salary.as_matrix())

fig, (ax1, ax2) = plt.subplots(1,2, figsize=(11,4))
ax1.hist(df.Salary.as_matrix())
ax1.set_xlabel('Salary')
ax2.hist(y)
ax2.set_xlabel('Log(Salary)');
regr = DecisionTreeRegressor(max_leaf_nodes=3)
regr.fit(X, y)<jupyter_output><empty_output><jupyter_text>Figure 8.1<jupyter_code>graph = print_tree(regr, features=['Years', 'Hits'])
Image(graph.create_png())<jupyter_output><empty_output><jupyter_text>Figure 8.2<jupyter_code>df.plot('Years', 'Hits', kind='scatter', color='orange', figsize=(7,6))
plt.xlim(0,25)
plt.ylim(ymin=-5)
plt.xticks([1, 4.5, 24])
plt.yticks([1, 117.5, 238])
plt.vlines(4.5, ymin=-5, ymax=250)
plt.hlines(117.5, xmin=4.5, xmax=25)
plt.annotate('R1', xy=(2,117.5), fontsize='xx-large')
plt.annotate('R2', xy=(11,60), fontsize='xx-large')
plt.annotate('R3', xy=(11,170), fontsize='xx-large');<jupyter_output>C:\ProgramData\Anaconda3\lib\site-packages\matplotlib\axes\_base.py:3604: MatplotlibDeprecationWarning: 
The `ymin` argument was deprecated in Matplotlib 3.0 and will be removed in 3.2. Use `bottom` instead.
  alternative='`bottom`', obj_type='argument')<jupyter_text>PruningThis is currently not supported in scikit-learn. See first point under 'disadvantages of decision trees in the documentation. Implementation has been discussed but Random Forests have better predictive qualities than a single pruned tree anyway if I understand correctly.      8.1.2 Classification Trees<jupyter_code>df2 = pd.read_csv('input/20190903/ISLR_Datasets/Heart.csv').drop('Unnamed: 0', axis=1).dropna()
df2.info()
df2.ChestPain = pd.factorize(df2.ChestPain)[0]
df2.Thal = pd.factorize(df2.Thal)[0]
X2 = df2.drop('AHD', axis=1)
y2 = pd.factorize(df2.AHD)[0]
clf = DecisionTreeClassifier(max_depth=None, max_leaf_nodes=6, max_features=3)
clf.fit(X2,y2)
clf.score(X2,y2)
graph2 = print_tree(clf, features=X2.columns, class_names=['No', 'Yes'])
Image(graph2.create_png())<jupyter_output><empty_output><jupyter_text>Lab  8.3.1 Fitting Classification Trees<jupyter_code>df3 = pd.read_csv('input/20190903/ISLR_Datasets/Carseats.csv').drop('Unnamed: 0', axis=1)
df3.head()
df3['High'] = df3.Sales.map(lambda x: 1 if x>8 else 0)
df3.ShelveLoc = pd.factorize(df3.ShelveLoc)[0]

df3.Urban = df3.Urban.map({'No':0, 'Yes':1})
df3.US = df3.US.map({'No':0, 'Yes':1})
df3.info()
df3.head(5)
X = df3.drop(['Sales', 'High'], axis=1)
y = df3.High

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)
clf = DecisionTreeClassifier(max_depth=6)
clf.fit(X, y)
print(classification_report(y, clf.predict(X)))
graph3 = print_tree(clf, features=X.columns, class_names=['No', 'Yes'])
Image(graph3.create_png())
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
cm = pd.DataFrame(confusion_matrix(y_test, pred).T, index=['No', 'Yes'], columns=['No', 'Yes'])
cm.index.name = 'Predicted'
cm.columns.name = 'True'
cm
# Precision of the model using test data is 74%
print(classification_report(y_test, pred))<jupyter_output>precision    recall  f1-score   support

           0       0.76      0.86      0.80       118
           1       0.75      0.61      0.67        82

   micro avg       0.76      0.76      0.76       200
   macro avg       0.75      0.73      0.74       200
weighted avg       0.75      0.76      0.75       200<jupyter_text>Pruning not implemented in scikit-learn.  8.3.2 Fitting Regression Trees<jupyter_code>boston_df = pd.read_csv('input/20190903/ISLR_Datasets/Boston.csv')
boston_df.info()
X = boston_df.drop('medv', axis=1)
y = boston_df.medv

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)
# Pruning not supported. Choosing max depth 3)
regr2 = DecisionTreeRegressor(max_depth=3)
regr2.fit(X_train, y_train)
pred = regr2.predict(X_test)
graph = print_tree(regr2, features=X.columns)
Image(graph.create_png())
plt.scatter(pred, y_test, label='medv')
plt.plot([0, 1], [0, 1], '--k', transform=plt.gca().transAxes)
plt.xlabel('pred')
plt.ylabel('y_test')
mean_squared_error(y_test, pred)<jupyter_output><empty_output><jupyter_text>8.3.3 Bagging and Random Forests<jupyter_code># There are 13 features in the dataset
X.shape
# Bagging: using all features
regr1 = RandomForestRegressor(max_features=13, random_state=1)
regr1.fit(X_train, y_train)
pred = regr1.predict(X_test)

plt.scatter(pred, y_test, label='medv')
plt.plot([0, 1], [0, 1], '--k', transform=plt.gca().transAxes)
plt.xlabel('pred')
plt.ylabel('y_test')
mean_squared_error(y_test, pred)
# Random forests: using 6 features
regr2 = RandomForestRegressor(max_features=6, random_state=1)
regr2.fit(X_train, y_train)
pred = regr2.predict(X_test)
mean_squared_error(y_test, pred)
Importance = pd.DataFrame({'Importance':regr2.feature_importances_*100}, index=X.columns)
Importance.sort_values('Importance', axis=0, ascending=True).plot(kind='barh', color='r', )
plt.xlabel('Variable Importance')
plt.gca().legend_ = None<jupyter_output><empty_output><jupyter_text>8.3.4 Boosting<jupyter_code>regr = GradientBoostingRegressor(n_estimators=500, learning_rate=0.01, random_state=1)
regr.fit(X_train, y_train)
feature_importance = regr.feature_importances_*100
rel_imp = pd.Series(feature_importance, index=X.columns).sort_values(inplace=False)
print(rel_imp)
rel_imp.T.plot(kind='barh', color='r', )
plt.xlabel('Variable Importance')
plt.gca().legend_ = None
mean_squared_error(y_test, regr.predict(X_test))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11516213847502191, 'fraction_numerical': 0.03137598597721297, 'mean_word_length': 4.91856846473029, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '456797', 'license': 'MIT', 'n_tokens_mistral': 3956, 'n_tokens_neox': 3685, 'n_words': 1145, 'path': 'Lectures/2019.09.03 Decision Tree Regressor.ipynb', 'repo_name': 'lev1khachatryan/ASDS_ML-2', 'size': '11417'}