<jupyter_start><jupyter_text>Decision Trees for Iris Species Classification[<NAME>](https://github.com/ccampguilhem), March 2018  Table of contents- Introduction- Iris species dataset- High-dimensional data visualization- Dimensionality reduction- What is machine learning ?- Modelling in reduced space- Conclusion  IntroductionThis notebook shows the capability of Decision Trees from [Scikit-Learn](http://scikit-learn.org/stable/index.html) package as well as some visualization techniques provided by [pandas](https://pandas.pydata.org/) and [seaborn](https://seaborn.pydata.org/). I have decided to use the [Iris dataset](https://www.kaggle.com/uciml/iris) to illustrate what we can do with it.The Iris Species dataset is a famous dataset for any machine learning amateur, as it's a simple dataset which can be used for classification problems. The Iris Species dataset was originally used in R.A. Fisher's classic 1936 [paper](http://rcs.chemometrics.ru/Tutorials/classification/Fisher.pdf), The Use of Multiple Measurements in Taxonomic Problems. I am following the Udadicty free Machine Learning [course](https://eu.udacity.com/course/machine-learning--ud262) which main problem is not to have assignments ! So this is an attempt to illustrate what I have learnt there.  Iris species dataset I use the Kaggle beta [API](https://github.com/Kaggle/kaggle-api) to collect the dataset on my local computer.First we download the dataset locally. If you want to replicate this step you need to install the Kaggle beta API and create an account on Kaggle website. Finally you will need to create an API token key for your account. The GitHub [repository](https://github.com/Kaggle/kaggle-api) has all information required.<jupyter_code>from kaggle.api import KaggleApi
connection = KaggleApi()
connection.authenticate()
#connection.datasetsList(search="iris")
#connection.datasetListFiles("uciml/iris")
connection.datasetDownloadFile("uciml/iris", "Iris.csv", ".")<jupyter_output><empty_output><jupyter_text>We can load the dataset with pandas:<jupyter_code>import pandas as pd
df = pd.read_csv("uciml/iris/Iris.csv")
df = df.set_index("Id")
df.info()
df.head()<jupyter_output><empty_output><jupyter_text>And get few statistics about the dataset:<jupyter_code>df.describe()
print df["Species"].value_counts(dropna=False)<jupyter_output>Iris-setosa        50
Iris-versicolor    50
Iris-virginica     50
Name: Species, dtype: int64<jupyter_text>The dataset is pretty clean. No null values, so we will keep the 150 samples as-is. For each sample we have 4 measures:- sepal length- sepal width- petal length- petal widthFor each sample we have a "class" which is the the type of iris. We have 50 samples for each type:- setosa- versicolor- virginica<jupyter_code>%matplotlib inline
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
sns.set_style("whitegrid")
plt.rcParams["figure.figsize"] = (14, 7)<jupyter_output><empty_output><jupyter_text>Convert to long format to ease distribution plots:<jupyter_code>df_melted = df.melt(id_vars=["Species"])
df_melted.head()<jupyter_output><empty_output><jupyter_text>We can start be having a look a distribution for each feature using a [boxplot](https://seaborn.pydata.org/generated/seaborn.boxplot.html):<jupyter_code>ax = sns.boxplot(x="variable", y="value", hue="Species", data=df_melted)
ax.set_title("Distribution of iris features", fontsize=20)
ax.set_xlabel("");<jupyter_output><empty_output><jupyter_text>We can see that distribution for petal length and width are quite different for the 3 different species, more specifically for Iris-setosa which is smaller. Sepal features bring "less" information on how to make distinction between the species. Another way to visualize distributions is to use [histograms](https://seaborn.pydata.org/generated/seaborn.distplot.htmlseaborn.distplot) layout in a [facet grid](https://seaborn.pydata.org/generated/seaborn.FacetGrid.htmlseaborn.FacetGrid):<jupyter_code>grid = sns.FacetGrid(df_melted, col="variable", hue="Species", col_wrap=2, sharey=False, 
                     sharex=False, aspect=2)
grid = grid.map(sns.distplot, "value", bins=5, kde=False)
grid.add_legend()
grid.fig.subplots_adjust(top=0.9)
grid.fig.suptitle("Distribution of iris features", fontsize=20);<jupyter_output><empty_output><jupyter_text>From the above plot, we confirm the difference of Iris-setosa for petal features. We can figure out if features are correlated using a [pair plot](https://seaborn.pydata.org/generated/seaborn.pairplot.htmlseaborn.pairplot):<jupyter_code>ax = sns.pairplot(df, hue="Species", size=1.5, aspect=1.4)
ax.fig.suptitle("Relations between features", fontsize=20)
ax.fig.subplots_adjust(top=0.9);<jupyter_output><empty_output><jupyter_text>Sepal length is correlated (linearly) with sepal width. It is also correlated with petal length and width except for Iris-setosa. Similarly petal length and width are correlated, except for Iris-setosa.  High-dimensional data visualizationVisualization of high-dimensional data is a challenge. Both pandas and scikit-learn bring options to the table. The [Andrews curves](https://en.wikipedia.org/wiki/Andrews_plot) is a technique using values of features to feed a Fourier series implemented in [pandas](https://pandas.pydata.org/pandas-docs/stable/visualization.htmlandrews-curves). Each sample in the dataset is then represented by a curve:<jupyter_code>from pandas.plotting import andrews_curves
ax = andrews_curves(df, "Species", color=["#3274A1", "#E1812C", "#3A923A"])
ax.set_title("Iris dataset with Andrews curves", fontsize=20)
ax.set_frame_on(False)
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)<jupyter_output><empty_output><jupyter_text>Once again, we can see that Iris-setosa clearly stands out again other species of Iris. Another way to visualize the dataset is to used samples projection on 2D space techniques. [Scikit-learn](http://scikit-learn.org/stable/modules/manifold.htmlmanifold-learning) comes with manifold techniques, for example t-distributed Stochastic Neighbor Embedding (t-SNE). This method may be computationally expensive in very high-dimension spaces.<jupyter_code>from sklearn.manifold import TSNE

df_features = df[["SepalLengthCm", "SepalWidthCm", "PetalLengthCm", 
                  "PetalWidthCm"]]
df_classes = df[["Species"]]

tsne = TSNE(n_components=2, random_state=1234)
df_proj = pd.DataFrame(tsne.fit_transform(df_features))
df_proj["Species"] = df_classes.Species.values
df_proj = df_proj.rename(columns={0: "var1", 1: "var2"})
df_proj.head()
ax = sns.lmplot(x="var1", y="var2", hue="Species", data=df_proj, size=6, aspect=1.5, fit_reg=False, 
               scatter_kws={"s": 100})
ax.fig.suptitle("Iris species with t-SNE manifold", fontsize=20)
ax.fig.subplots_adjust(top=0.9);
ax.fig.axes[0].get_xaxis().set_visible(False)
ax.fig.axes[0].get_yaxis().set_visible(False)
ax.fig.axes[0].set_frame_on(False)<jupyter_output><empty_output><jupyter_text>For more information on t-SNE check this [video](https://www.youtube.com/watch?v=NEaUSP4YerM).  Dimensionality reductionFor such a simple problem (4 features), it is not necessary to reduce dimensionality for training an algorithm. However, it will come in handy to visualize the boundaries created by our decision tree classifier. We are going to keep only two dimensions for the sake of visualization simplicity. We will use a principal component analysis decomposition (PCA) from [sklearn](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.htmlsklearn.decomposition.PCA) package.<jupyter_code>from sklearn.decomposition import PCA
pca = PCA(n_components=2)

pca.fit(df_features)
df_reduced = pd.DataFrame(pca.transform(df_features))
df_reduced["Species"] = df_classes.Species.values
df_reduced = df_reduced.rename(columns={0: "var1", 1: "var2"})
df_reduced.head()<jupyter_output><empty_output><jupyter_text>After PCA algorithm has been applied, we can see on this reduced dimension space that Iris-setosa is pretty easy to identify. The boundary between Iris-versicolor and Iris-virginica is less obvious:<jupyter_code>ax = sns.lmplot(x="var1", y="var2", hue="Species", data=df_reduced, size=6, aspect=1.5, fit_reg=False, 
               scatter_kws={"s": 100})
ax.fig.suptitle("Iris species in reduced dimension space", fontsize=20)
ax.fig.subplots_adjust(top=0.9);<jupyter_output><empty_output><jupyter_text>What is machine learning ?![Machine Learning](./img/machinelearning.jpg)Source: https://www.saagie.com/fr/blog/machine-learning-pour-les-grand-meres  Supervised Learning:![Machine Learning](./img/classification_vs_regression.png)**Classification**: what is the type ?**Regression**: what is the value ?Source: https://medium.com/@heyozramos/regression-vs-classification-86d73c281c5e  Our problem statement (supervised classification):Given measures of Iris, what species is that ?  Methodology- Use a subset of dataset to **train** an algorithm- Use the remaining subset to **test** algorithm  Modelling in reduced spaceIn this section, we are going to train a [decision tree classifier](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html). First wee need to split the dataset into training and testing datasets. We will train the algorithm in the reduced space that we have created above with PCA.  Split training and testing datasetWe use 30% percent of dataset for test. I am using [StratifiedShuffleSplit](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.htmlsklearn.model_selection.StratifiedShuffleSplit) to keep the same proportion for each species in the training and testing datasets.<jupyter_code>from sklearn.model_selection import StratifiedShuffleSplit
splitter = StratifiedShuffleSplit(n_splits=1, test_size=0.3, random_state=1234)
indices = splitter.split(df_features, df_classes).next()
features_train = df_reduced.iloc[indices[0], [0, 1]]
targets_train = df_reduced.iloc[indices[0], [2]]
features_test = df_reduced.iloc[indices[1], [0, 1]]
targets_test = df_reduced.iloc[indices[1], [2]]
def plot_dataset(features_train, targets_train, features_test, targets_test, clf=None, title=None):
    """
    Plot train and test datasets on given Axes
    
    - features_train: training features
    - targets_train: training classes
    - features_test: testing features
    - targets_test: testing classes
    - ax: matplotlib Axes object
    - clf: trained classifier
    """
    #Create figure
    fig = plt.figure()
    ax = fig.add_subplot(111)
    #Classifier boundary
    if clf is not None:
        #Create a grid
        x, y = np.meshgrid(np.linspace(-4, 4, 30), np.linspace(-1.5, 1.5, 30))
        X = x.flatten()
        Y = y.flatten()
        inp = np.vstack((X, Y)).transpose()
        #Predict classes
        pred = clf.predict(inp)
        #Convert to numerical values
        dct = {"Iris-setosa": 0, "Iris-versicolor": 1, "Iris-virginica": 2}
        dctinv = {0: "Iris-setosa", 1: "Iris-versicolor", 2: "Iris-virginica"}
        f = lambda x: an[x]
        conv = np.vectorize(lambda x: dct[x])
        Z = conv(pred)
        #Plot contour
        z = Z.reshape(x.shape)
        levels = [-0.5, 0.5, 1.5, 2.5]
        cf = ax.contourf(x, y, z, levels=levels, colors=["#3274A1", "#E1812C", "#3A923A"], alpha=0.5)
        cbar = fig.colorbar(cf, ax=ax, orientation="horizontal", aspect=20, fraction=0.07, shrink=1)
        cbar.set_ticks([0,1,2])
        cbar.set_ticklabels([dctinv[t] for t in [0,1,2]])
    #Display dataset
    for (species, color) in [("Iris-setosa", "#3274A1"), ("Iris-versicolor", "#E1812C"), ("Iris-virginica", "#3A923A")]:
        train = features_train[targets_train.Species == species]
        test = features_test[targets_test.Species == species]
        label_train = "{} ({})".format(species, "Train")
        label_test = "{} ({})".format(species, "Test")
        ax.scatter(x=train["var1"], y=train["var2"], color=color, marker="o", s=200, label=label_train, 
                   edgecolors="#ffffff")
        ax.scatter(x=test["var1"], y=test["var2"], color=color, marker="^", s=200, label=label_test, 
                  edgecolors="#ffffff")
    #Plot configuration
    if title is None:
        ax.set_title("Train and test dataset", fontsize=20)
    else:
        ax.set_title(title, fontsize=20)
    ax.set_xlabel("var1")
    ax.set_ylabel("var2")
    ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0., frameon=True);    
    return ax<jupyter_output><empty_output><jupyter_text>Let's visualize the split that has been operated:<jupyter_code>plot_dataset(features_train, targets_train, features_test, targets_test);<jupyter_output><empty_output><jupyter_text>We can see that for each species, test samples have been randomized to cover the space occupied by that species.  Train a decision treeWe are going to make the following assumption: the dataset is linearly separable. We then restrict the algorithm to only draws a very simple boundary. We can achieve this by limiting the maximum depth of decision tree to 1.<jupyter_code>from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(random_state=1234, max_depth=1)
clf.fit(features_train, targets_train)<jupyter_output><empty_output><jupyter_text>A linear boundary (background color) is calculated by the algorithm, which enables to identify Iris-setosa but not the other species:<jupyter_code>plot_dataset(features_train, targets_train, features_test, targets_test, clf, title="Highly biassed prediction");<jupyter_output><empty_output><jupyter_text>Now reduce the bias by increasing the maximum depth of decision tree to 2. The algorithms does a better job but may still be improved:<jupyter_code>clf = DecisionTreeClassifier(random_state=1234, max_depth=2)
clf.fit(features_train, targets_train)
plot_dataset(features_train, targets_train, features_test, targets_test, clf, title="Imperfect prediction");<jupyter_output><empty_output><jupyter_text>Increasing the maximum depth to 4 shows very interesting results even if few samples are incorrectly predicted:<jupyter_code>clf = DecisionTreeClassifier(random_state=1234, max_depth=4)
clf.fit(features_train, targets_train)
plot_dataset(features_train, targets_train, features_test, targets_test, clf, title="Best predicition");
#Export the decision to Graphviz format
from sklearn.tree import export_graphviz
export_graphviz(clf, "tree.dot", rounded=True, impurity=False, 
                feature_names=["var1", "var2"], class_names=["Iris-setosa", "Iris-versicolor", "Iris-virginica"])
!dot -Tpng tree.dot -o img/tree.png<jupyter_output><empty_output><jupyter_text>Increasing the maximum depth to 6 shows that decision tree tries to get the previously incorrectly predicted samples. This increases the variance of the model, it becomes very sensible to "exceptional" values. This is a situation of over-fitting that should be avoided:<jupyter_code>clf = DecisionTreeClassifier(random_state=1234, max_depth=6)
clf.fit(features_train, targets_train)
plot_dataset(features_train, targets_train, features_test, targets_test, clf, title="Over-fitted prediction");<jupyter_output><empty_output><jupyter_text>A decision tree with a maximum depth of 4 is probably the best trade-off between:- **bias**: errors from initial assumptions in algorithm- **variance**: errors due to sensitivity to small fluctuations in training datasetA good predictive model has low bias and low variance. Here is our decision trained tree: Another way to measure accuracy of a predictive model is to use metrics:<jupyter_code>from sklearn.metrics import precision_recall_fscore_support

clf = DecisionTreeClassifier(random_state=1234, max_depth=4)
clf.fit(features_train, targets_train)
targets_pred = clf.predict(features_test)

precision, recall, fscore, support = precision_recall_fscore_support(
    targets_test, targets_pred)
df_metrics = pd.DataFrame({"precision": precision, 
                           "recall": recall, 
                           "fscore": fscore, 
                           "Species": ["Iris-setosa", "Iris-versicolor", "Iris-virginica"]})
df_metrics.set_index("Species", inplace=True)<jupyter_output><empty_output><jupyter_text>- **precision**: prediction is consistent with the actual class of test sample- **recall**: capability to correctly predict all items of a given class- **fscore**: averaged score between precision and recall<jupyter_code>df_metrics.head()<jupyter_output><empty_output><jupyter_text>Here, we have one Iris-versicolor which has been predicted as a Iris-virginica. This decreases the **precision** of Iris-virginica and the **recall** of Iris-versicolor.  Conclusion- pandas and seaborn provides usefull and easy-to-use capabilities to **see** the data, even in high-dimensions.- Tempted by marchine learning ? Start with **decision trees** and avoid complex alternatives (neural networks...).- **Always** keep samples to test accurary of your predictive model.- Consider **dimension reduction** to better understand the behavior of your predictive model.- Tune parameters to reach a **low-bias** and a **low-variance** model.- Someone claims he has incredible predictive model ? Ask he/she the **precision** and the **recall** scores on the **test** samples, just in case...  Appendix t-SNE algorithm explained by [StatQuest](https://www.youtube.com/watch?v=NEaUSP4YerM)<jupyter_code>#Convert notebook to html
!jupyter nbconvert --to html --template html_minimal.tpl --no-prompt iris_dataset_classification.ipynb
#Convert notebook to slide show
!jupyter nbconvert --to slides --template html_slides_minimal --no-prompt iris_dataset_classification.ipynb<jupyter_output>[NbConvertApp] Converting notebook iris_dataset_classification.ipynb to slides
[NbConvertApp] Writing 256548 bytes to iris_dataset_classification.slides.html
{'fraction_non_alphanumeric': 0.10738480488214545, 'fraction_numerical': 0.01455685571916466, 'mean_word_length': 5.429805615550756, 'pattern_counts': {'":': 9, '<': 88, '<?xml version=': 0, '>': 88, 'https://': 17, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '582709', 'license': 'MIT', 'n_tokens_mistral': 5784, 'n_tokens_neox': 5459, 'n_words': 1616, 'path': 'M01-SupervisedLearning/SL01-DecisionTrees/iris_dataset_classification.ipynb', 'repo_name': 'ccampguilhem/Udacity-MachineLearning', 'size': '17873'}