<jupyter_start><jupyter_text>*This notebook contains an excerpt from the upcoming book [Machine Learning for OpenCV](https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-opencv) by <NAME> (expected Aug 2017).The code is released under the [MIT license](https://opensource.org/licenses/MIT),and is available on [GitHub](https://github.com/mbeyeler/opencv-machine-learning).**Note that this excerpt contains only the raw code - the book is rich with additional explanations and illustrations.If you find this content useful, please consider supporting the work by[buying the book](https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-opencv)!*   Building your first decision treeLet's consider the example where a number of patients have suffered from the same illness - such as a rare form of basorexia. Let's further assume that the true causes of the disease remain unknown to this day, and that all the information that is available to us consists of a bunch of physiological measurements. For example, we might have access to the following information:- a patient's blood pressure ('BP'),- a patient's cholesterol level ('cholesterol'),- a patient's gender ('sex'),- a patient's age ('age'),- a patient's blood sodium concentration ('Na'), and- a patient's blood potassium concentration ('K').Based on all this information, let's suppose a doctor made recommendations for the patient to treat their disease using one of four possible drugs - drug A, B, C, or D. We have data available for 20 different patients:<jupyter_code>data = [
    {'age': 33, 'sex': 'F', 'BP': 'high', 'cholesterol': 'high', 'Na': 0.66, 'K': 0.06, 'drug': 'A'},
    {'age': 77, 'sex': 'F', 'BP': 'high', 'cholesterol': 'normal', 'Na': 0.19, 'K': 0.03, 'drug': 'D'},
    {'age': 88, 'sex': 'M', 'BP': 'normal', 'cholesterol': 'normal', 'Na': 0.80, 'K': 0.05, 'drug': 'B'},
    {'age': 39, 'sex': 'F', 'BP': 'low', 'cholesterol': 'normal', 'Na': 0.19, 'K': 0.02, 'drug': 'C'},
    {'age': 43, 'sex': 'M', 'BP': 'normal', 'cholesterol': 'high', 'Na': 0.36, 'K': 0.03, 'drug': 'D'},
    {'age': 82, 'sex': 'F', 'BP': 'normal', 'cholesterol': 'normal', 'Na': 0.09, 'K': 0.09, 'drug': 'C'},
    {'age': 40, 'sex': 'M', 'BP': 'high', 'cholesterol': 'normal', 'Na': 0.89, 'K': 0.02, 'drug': 'A'},
    {'age': 88, 'sex': 'M', 'BP': 'normal', 'cholesterol': 'normal', 'Na': 0.80, 'K': 0.05, 'drug': 'B'},
    {'age': 29, 'sex': 'F', 'BP': 'high', 'cholesterol': 'normal', 'Na': 0.35, 'K': 0.04, 'drug': 'D'},
    {'age': 53, 'sex': 'F', 'BP': 'normal', 'cholesterol': 'normal', 'Na': 0.54, 'K': 0.06, 'drug': 'C'},
    {'age': 36, 'sex': 'F', 'BP': 'high', 'cholesterol': 'high', 'Na': 0.53, 'K': 0.05, 'drug': 'A'},
    {'age': 63, 'sex': 'M', 'BP': 'low', 'cholesterol': 'high', 'Na': 0.86, 'K': 0.09, 'drug': 'B'},
    {'age': 60, 'sex': 'M', 'BP': 'low', 'cholesterol': 'normal', 'Na': 0.66, 'K': 0.04, 'drug': 'C'},
    {'age': 55, 'sex': 'M', 'BP': 'high', 'cholesterol': 'high', 'Na': 0.82, 'K': 0.04, 'drug': 'B'},
    {'age': 35, 'sex': 'F', 'BP': 'normal', 'cholesterol': 'high', 'Na': 0.27, 'K': 0.03, 'drug': 'D'},
    {'age': 23, 'sex': 'F', 'BP': 'high', 'cholesterol': 'high', 'Na': 0.55, 'K': 0.08, 'drug': 'A'},
    {'age': 49, 'sex': 'F', 'BP': 'low', 'cholesterol': 'normal', 'Na': 0.27, 'K': 0.05, 'drug': 'C'},
    {'age': 27, 'sex': 'M', 'BP': 'normal', 'cholesterol': 'normal', 'Na': 0.77, 'K': 0.02, 'drug': 'B'},
    {'age': 51, 'sex': 'F', 'BP': 'low', 'cholesterol': 'high', 'Na': 0.20, 'K': 0.02, 'drug': 'D'},
    {'age': 38, 'sex': 'M', 'BP': 'high', 'cholesterol': 'normal', 'Na': 0.78, 'K': 0.05, 'drug': 'A'}
]<jupyter_output><empty_output><jupyter_text>Understanding the task by understanding the dataWhat is always the first step in tackling a new machine learning problem?You are absolutely right: to get a sense of the data. The better you understand the data, the better you understand the problem you are trying to solve.The first thing to realize is that the 'drug' column is actually not a feature value like all the other columns. Thus, it would be a good idea to remove all 'drug' entries from the dictionaries listed in the 'data' variable and store them in a separate variable. For this we need to go through the list and extract the 'drug' entry, which is easiest to do with a list comprehension:<jupyter_code>target = [d['drug'] for d in data]
target<jupyter_output><empty_output><jupyter_text>Then remove the 'drug' entry from all the dictionaries:<jupyter_code>[d.pop('drug') for d in data];<jupyter_output><empty_output><jupyter_text>Sweet! Now let's look at the data:<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
age = [d['age'] for d in data]
age
sodium = [d['Na'] for d in data]
potassium = [d['K'] for d in data]
plt.figure(figsize=(10, 6))
plt.scatter(sodium, potassium)
plt.xlabel('sodium')
plt.ylabel('potassium')<jupyter_output><empty_output><jupyter_text>But, what we really want is to color the data points according to their target labels:<jupyter_code>target = [ord(t) - 65 for t in target]
target
plt.figure(figsize=(14, 10))

plt.subplot(221)
plt.scatter([d['Na'] for d in data], [d['K'] for d in data],
            c=target, s=100)
plt.xlabel('sodium (Na)')
plt.ylabel('potassium (K)')

plt.subplot(222)
plt.scatter([d['age'] for d in data], [d['K'] for d in data],
            c=target, s=100)
plt.xlabel('age')
plt.ylabel('potassium (K)')

plt.subplot(223)
plt.scatter([d['age'] for d in data], [d['Na'] for d in data],
            c=target, s=100)
plt.xlabel('age')
plt.ylabel('sodium (Na)')
plt.savefig('all.png')<jupyter_output><empty_output><jupyter_text>Preprocessing the dataWe need to convert all categorical features into numerical features:<jupyter_code>from sklearn.feature_extraction import DictVectorizer
vec = DictVectorizer(sparse=False)
data_pre = vec.fit_transform(data)
vec.get_feature_names()
data_pre[0]<jupyter_output><empty_output><jupyter_text>Convert to 32-bit floating point numbers in order to make OpenCV happy:<jupyter_code>import numpy as np
data_pre = np.array(data_pre, dtype=np.float32)
target = np.array(target, dtype=np.float32).reshape((-1, 1))
data_pre.shape, target.shape<jupyter_output><empty_output><jupyter_text>Then split data into training and test sets:<jupyter_code>import sklearn.model_selection as ms
X_train, X_test, y_train, y_test = ms.train_test_split(data_pre, target, test_size=5, random_state=42)<jupyter_output><empty_output><jupyter_text>Building the decision treeBuilding the decision tree with OpenCV works in much the same way as in Chapter 3, First Steps in Supervised Learning. Recall that all machine learning function reside in OpenCV 3.1's ml module. You can create an empty decision tree using the following code:<jupyter_code>import cv2
dtree = cv2.ml.DTrees_create()<jupyter_output><empty_output><jupyter_text>Then train the model:<jupyter_code>dtree.train(X_train, cv2.ml.ROW_SAMPLE, y_train)<jupyter_output><empty_output><jupyter_text>Predict some values:<jupyter_code>y_pred = dtree.predict(X_test)<jupyter_output><empty_output><jupyter_text>Calculate the score on the training and test sets:<jupyter_code>from sklearn import metrics
metrics.accuracy_score(y_test, dtree.predict(X_test))
metrics.accuracy_score(y_train, dtree.predict(X_train))<jupyter_output><empty_output><jupyter_text>Visualizing a trained decision treeOpenCV's implementation of decision trees is good enough if you are just starting out, and don't care too much what's going on under the hood. However, in the following sections we will switch to Scikit-Learn. Their implementation allows us to customize the algorithm and makes it a lot easier to investigate the inner workings of the tree. Their usage is also much better documented...<jupyter_code>from sklearn import tree
dtc = tree.DecisionTreeClassifier()<jupyter_output><empty_output><jupyter_text>The model is trained by calling `fit`:<jupyter_code>dtc.fit(X_train, y_train)
dtc.score(X_train, y_train)
dtc.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Now, here's the cool thing: If you want to know what the tree looks like, you can do so using GraphViz to create a PDF file (or any other supported file type) from the tree structure. For this to work, you need to install GraphViz first, which you can do from the command line using conda:    $ conda install graphviz<jupyter_code>with open("tree.dot", 'w') as f:
    f = tree.export_graphviz(dtc, out_file=f,
                            feature_names=vec.get_feature_names(),
                            class_names=['A', 'B', 'C', 'D'])<jupyter_output><empty_output><jupyter_text>Then, back on the command line, you can use GraphViz to turn "tree.dot" into (for example) a PNG file:    $ dot -Tpng tree.dot -o tree.png  Rating the importance of featuresScikit-Learn provides a function to rate feature importance, which is a number between 0 and 1 for each feature, where 0 means "not used at all in any decisions made" and 1 means "perfectly predicts the target". The feature importances are normalized such that they all sum to 1:<jupyter_code>dtc.feature_importances_<jupyter_output><empty_output><jupyter_text>If we remind ourselves of the feature names, it will become clear which features seem to be the most important. A plot might be most informative:<jupyter_code>plt.figure(figsize=(12, 6))
plt.barh(range(10), dtc.feature_importances_, align='center', tick_label=vec.get_feature_names())
plt.savefig('feature_imp.png')<jupyter_output><empty_output><jupyter_text>Understanding decision rulesTwo of the most commonly used criteria for making decisions are the following:- `criterion='gini'`: The Gini impurity is a measure of misclassification, with the aim of minimizing the probability of misclassification.- `criterion='entropy'`: In information theory, entropy is a measure of the amount of uncertainty associated with a signal or distribution.In Scikit-Learn, you can specify the split criterion in the constructor of the decision tree call. For example, if you want to use entropy, you would type the following:<jupyter_code>dtce = tree.DecisionTreeClassifier(criterion='entropy')
dtce.fit(X_train, y_train)
dtce.score(X_train, y_train)
dtce.score(X_test, y_test)
with open("tree.dot", 'w') as f:
    f = tree.export_graphviz(dtce, out_file=f,
                            feature_names=vec.get_feature_names(),
                            class_names=['A', 'B', 'C', 'D'])<jupyter_output><empty_output><jupyter_text>Controlling the complexity of decision treesThere are two common ways to avoid overfitting:- pre-pruning: This is the process of stopping the creation of the tree early.- post-pruning (or just pruning): This is the process of first building the tree but then removing or collapsing nodes that contain only little information.There are a number of ways to pre-prune a tree, all of which can be achieved by passing optional arguments to the `DecisionTreeClassifier` constructor:- limiting the maximum depth of the tree via the `max_depth parameter`- limiting the maximum number of leaf nodes via `max_leaf_nodes`- requiring a minimum number of points in a node to keep splitting it via `min_samples_split`<jupyter_code>dtc0 = tree.DecisionTreeClassifier(criterion='entropy', max_leaf_nodes=6)
dtc0.fit(X_train, y_train)
dtc0.score(X_train, y_train)
dtc0.score(X_test, y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14461942257217847, 'fraction_numerical': 0.019772528433945755, 'mean_word_length': 5.336474501108648, 'pattern_counts': {'":': 0, '<': 78, '<?xml version=': 0, '>': 78, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '180359', 'license': 'MIT', 'n_tokens_mistral': 4030, 'n_tokens_neox': 3694, 'n_words': 1327, 'path': 'resources/Machine Learning for OpenCV/Chapter05/05.01-Building-Our-First-Decision-Tree.ipynb', 'repo_name': 'arbbakbenny/awesome-AI-books', 'size': '11439'}