<jupyter_start><jupyter_text>Decision trees Decision trees are a great way to get our feet wet with machine learning, because they are as transparent or white-box as can be. This means that they rank very high on explainability. In fact, you can see that no explanation is required, becaues the tree itself is the explanation. No equations needed!Let's read our tennis data set. As we'll later see, it's the kind of data set that works especially well for tree-based models: the features and target are both categorical with few categories.<jupyter_code>import pandas as pd

tennis = pd.read_csv("data/tennis.csv")
tennis.head()<jupyter_output><empty_output><jupyter_text>And the first thing we will do is split our data into training and test sets. In this case, with the data being so small, the test set will be very limited. We will address this issue later.<jupyter_code>from sklearn.model_selection import train_test_split 

X = tennis.drop(columns = "play")
Y = tennis["play"]
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.1, random_state = 99)
print(y_test)<jupyter_output><empty_output><jupyter_text>ExerciseNote that `DecisionTreeClassifier` just like most other algorithms in `sklearn` don't directly work with categorical data: which includes `object` and `category` types. Uncomment the third line of the following code and run it to check the error message you get.<jupyter_code>from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(criterion = 'entropy', max_depth = 3)
# tree.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>End of exercise So we need to turn our categorical columns into numeric ones, using methods we learned earlier in our discussion about feature engineering. One option is to use `LabelEncoder`, which **remaps** or **recodes** the categories in a categorical column **into integers**.<jupyter_code>from sklearn.preprocessing import LabelEncoder
X_train_remap = X_train.copy()
X_test_remap = X_test.copy()

for col in X_train_remap.columns:
    le = LabelEncoder()
    le.fit(X_train[col])
    X_train_remap[col] = le.transform(X_train[col])
    X_test_remap[col] = le.transform(X_test[col])
    # we can use this to print the mapping if we need to
    label_mapping = dict(zip(le.classes_, le.transform(le.classes_)))
    print(label_mapping)<jupyter_output><empty_output><jupyter_text>IMPORTANT: Notice that in the code above we run `fit` only **once** on the training data, but run `transform` **twice**: once on the training data and once on the test data.Here's what the original data looks like.<jupyter_code>X_train.head()<jupyter_output><empty_output><jupyter_text>Here's what `LabelEncoder` did with the data.<jupyter_code>X_train_remap.head()<jupyter_output><empty_output><jupyter_text>Let's now see if we can train a decision tree.<jupyter_code>tree_labelencoder = DecisionTreeClassifier(criterion = 'entropy', max_depth = 3)
tree_labelencoder.fit(X_train_remap, y_train)<jupyter_output><empty_output><jupyter_text>This time we don't get an error. So using `LabelEncoder` did the trick. How how good is our tree at predicting? Let's get predictions on the test data.<jupyter_code>y_pred_test = tree_labelencoder.predict(X_test_remap)<jupyter_output><empty_output><jupyter_text>And let's compare the predictions with the labels on the test data.<jupyter_code>from sklearn.metrics import confusion_matrix
confusion_matrix(y_test, y_pred_test)<jupyter_output><empty_output><jupyter_text>Two data points is not enough to judge a model's accuracy, but we will get to that later. How about visalizing the tree? We have a function for doing that. Make sure to change the path in the function below to point to the location of the `graphviz` library on your computer.<jupyter_code>import os
from sklearn import tree
from IPython.display import Image
graphviz_path = 'C:/Users/sethmott/AppData/Local/Continuum/anaconda3/pkgs/graphviz-2.38.0-4/Library/bin/graphviz'

def show_tree(out_name, model, feature_names):
    dotfile = open(out_name + ".dot", 'w')
    tree.export_graphviz(model, out_file = dotfile, filled = True, rounded = True, feature_names = feature_names)
    dotfile.close()

    os.environ["PATH"] += os.pathsep + graphviz_path
    os.system('dot -Tpng {}.dot -o {}.png'.format(out_name, out_name))

    return(Image(filename = out_name + '.png', width = 500, height = 500))
show_tree('tennis_tree_labelencoder', tree_labelencoder, X_train_remap.columns)<jupyter_output><empty_output><jupyter_text>Take a good look at the tree and make sure you are comfortable with what it's saying. It's supposed to be "white-box" but is it? Something about the tree is a little **unnatural**: when we used `LabelEncoder` it assigned integers to each category of each feature and did it in the **alphabetical order** of the categories' names. If that choice seems arbitrary, it's because it is. By assigning integers to categories, we are implying that there is some sort of ordering. In our case this is fine, because our categorical features are **ordinal** meaning that there is a natural order: `sunny` > `overcast` > `rainy` for `outlook`, `hot` > `mild` > `cool` for `temp`, etc. But the alphabetical ordering used by `LabelEncoder` means that for example `mild` > `hot` > `cool`, which distorts the order. As a result a split like `outlook <= 1.5` is hard to interpret. As a general rule, encoding categorical columns like this is not a good idea and at best should only be done with **ordinal** data in a way that respects the order. With **nominal** data this should be avoided altogether. Instead we should turn to one-hot encoding.<jupyter_code>from sklearn.preprocessing import OneHotEncoder
onehot = OneHotEncoder(sparse = False, handle_unknown = "ignore")
onehot.fit(X_train)<jupyter_output><empty_output><jupyter_text>Recall that `sklearn` transformations don't return nice-looking `DataFrame` objects. Instead they return `numpy` arrays, so it's up to us to convert the arrays back into `DataFrame` if we need to. In **production** we usually wouldn't need to do that, since we can pass the arrays directly to the ML algorithm for training and for scoring. But during **development** it can help us trace our steps and see intermediate results as we work toward the final product.<jupyter_code>onehot_col_names = onehot.get_feature_names(X_train.columns)
X_train_onehot = pd.DataFrame(onehot.transform(X_train), columns = onehot_col_names)
X_test_onehot = pd.DataFrame(onehot.transform(X_test), columns = onehot_col_names)

X_train_onehot.head()<jupyter_output><empty_output><jupyter_text>Generate and Evaluate the Model<jupyter_code>tree_onehot_ent = DecisionTreeClassifier(criterion = 'entropy', max_depth = 3)
tree_onehot_ent.fit(X_train_onehot, y_train)
y_pred_test = tree_onehot_ent.predict(X_test_onehot)
y_pred_test
y_test<jupyter_output><empty_output><jupyter_text>Let's look at our confusion matrix.<jupyter_code>confusion_matrix(y_test, y_pred_test),<jupyter_output><empty_output><jupyter_text>What is different about our tree now? The big change is that because of one-hot encoding, each split of the tree is now a **binary split** happening at one of the categories: for example `outlook_sunny <= 0.5` means that if `outlook` is not `sunny` then we move to the left (True always moves to the left), otherwise we move to the right (False always moves to the left).<jupyter_code>show_tree('tennis_tree_onehot_ent', tree_onehot_ent, X_train_onehot.columns)<jupyter_output><empty_output><jupyter_text>By default, the tree we used relied on **entropy** for its splitting criterion. Let's now see what happens if we use **Gini impurity** instead.<jupyter_code>tree_onehot_gini = DecisionTreeClassifier(criterion = 'gini')
tree_onehot_gini.fit(X_train_onehot, y_train)

y_pred_test = tree_onehot_gini.predict(X_test_onehot)<jupyter_output><empty_output><jupyter_text>Let's get the confusion matrix again.<jupyter_code>confusion_matrix(y_test, y_pred_test)<jupyter_output><empty_output><jupyter_text>And finally let's plot the tree.<jupyter_code>show_tree('tennis_tree_onehot_gini', tree_onehot_gini, X_train_onehot.columns)<jupyter_output><empty_output><jupyter_text>So far we trained three decision tree models, but with the test data consisting of only two rows, it's hard to evaluate the results and decide which model is better. What do we do in such situations? Depriving the training data of more data so we can have a bigger test data is definitely not a good solution. A better one is to just generate our own test data and try to simulate different situations and labeling it based on what we expect the label to be (we should be confident about our labels) and then predicting and evaluating the different models on this data. Of course generating such a data and making sure that it's representative of the training data is not always an easy task.  ExerciseIt's time to put to practice everything that we learned. Let's read a new data set.<jupyter_code>titanic = pd.read_csv('data/titanic.csv', sep = ',', header = 0)
titanic.columns = titanic.columns.str.lower() # use all lower case column names
titanic.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08178155286943264, 'fraction_numerical': 0.0026135249918327343, 'mean_word_length': 6.075500770416024, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 83, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '87680', 'license': 'MIT', 'n_tokens_mistral': 2837, 'n_tokens_neox': 2744, 'n_words': 1039, 'path': 'lesson_5.ipynb', 'repo_name': 'sethmott/datasci420', 'size': '9183'}