<jupyter_start><jupyter_text>Model Building and Evaluation - Load the engineered features and split into training and testing datasets- Search for the best parameters for the RandomForestClassifier- Display the results of the fitted classifier- Export the classifier for use elsewhere --- Code Import Necessary Tools and Libraries<jupyter_code>import sys
import os
sys.path.append(os.environ.get('NOTEBOOK_ROOT'))

import datetime
import itertools
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd 

from pprint import pprint
from sklearn.ensemble import RandomForestClassifier
from sklearn.externals import joblib
from sklearn.metrics import confusion_matrix
from sklearn.metrics import f1_score
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>Import DataImport the features list that we exported in the previous notebook.<jupyter_code>def get_features(path='engineered_features.csv'):
    """Gets and formats the engineered features from a CSV file.
    
    Args:
        path (string): The path of the CSV file.
        
    Returns:
        A dictionary with the features list and the labels list for the features.
    """
    data = pd.read_csv(path)    
    column_names = list(data)    
    
    FEATURES = ['red','green','blue','nir','swir1','swir2', 
                'NDVI',
                'NDVI_coeff_var',
                'PNDVI',
                'NBR',
                'NBR2',
                'NDWI_2',
                'SCI',
                'CVI',
                'CCCI',
                'bs','pv','npv'
               ]
    
    LABEL = ["LandUse"]
    
    features = data[FEATURES]
    labels = data[LABEL]
    return {'features': features, 'labels': labels}<jupyter_output><empty_output><jupyter_text>Split the FeaturesSplit the given features and labels into training and testing sets.<jupyter_code>def split_features(feature_set):
    """Splits the features and labels into training and test sets.
    
    Args:
        feature_set (dict): A dictionary of the features list and the labels. Needs a features and labels key.
    
    Returns:
        A dictionary of the training and testing sets.
    """
    features = feature_set['features']
    labels = feature_set['labels']
    
    X_train, X_test, y_train, y_test = train_test_split(
        features.values, labels.values.reshape(len(features))
    )
    return {'X_train': X_train, 'X_test': X_test, 'y_train': y_train, 'y_test': y_test}<jupyter_output><empty_output><jupyter_text>Search FunctionsFunctions for running a `GridSearchCV` and `RandomizedSearchCV` with specific search parameters over a `RandomForestClassifier`.<jupyter_code>def grid_search(feature_set):
    """Performs a grid search over a RandomForestClassifier.
    
    Args:
        feature_set (dict): The split feature set to run the search with. Needs a X_train and y_train key.
    
    Returns:
        The best estimated RandomForestClassifier object.
    """
    X_train = feature_set['X_train']
    y_train = feature_set['y_train']
    
    rf = RandomForestClassifier(random_state = 42)
    
    search_params = {'criterion': ['gini', 'entropy'],
                     'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None],
                     'min_samples_leaf': [1, 2, 4],
                     'min_samples_split': [2, 5, 10],
                     'n_estimators': [100, 200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000]
                    }

    grid_search = GridSearchCV(estimator=rf,
                               param_grid=search_params, 
                               cv=3,
                               n_jobs=-1,
                               verbose=0
                              )
    
    grid_search.fit(X_train, y_train)
    return grid_search.best_estimator_

def random_search(feature_set, n_iter):
    """Performs a grid search over a RandomForestClassifier.
    
    Args:
        feature_set (dict): The split feature set to run the search with. Needs a X_train and y_train key.
        n_iter (int): The number of iterations to search over.
    
    Returns:
        The best estimated RandomForestClassifier object.
    """
    X_train = feature_set['X_train']
    y_train = feature_set['y_train']
    
    rf = RandomForestClassifier(random_state = 42)
    
    search_params = {'criterion': ['gini', 'entropy'],
                     'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None],
                     'min_samples_leaf': [1, 2, 4],
                     'min_samples_split': [2, 5, 10],
                     'n_estimators': [100, 190, 280, 370, 460, 550, 640, 730, 820, 910, 1000]
                    }

    random_search = RandomizedSearchCV(estimator=rf,
                                       param_distributions=search_params,
                                       n_iter=n_iter,
                                       cv=3,
                                       n_jobs=-1,
                                       verbose=0
                                      )
    
    random_search.fit(X_train, y_train)
    return random_search.best_estimator_<jupyter_output><empty_output><jupyter_text>Display Importances PlotThis will display the important features as determined by the classifier.<jupyter_code>def importances_plot(rf, X):
    """Plots the important features of a RandomForestClassifier.
    
    Args:
        rf (RandomForestClassifier): The classifier object to plot the feature importances for.
        X: The values of the features used.

    Citiation:
    https://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html
    """
    FEATURES = ['red','green','blue','nir','swir1','swir2', 
                'NDVI',
                'NDVI_coeff_var',
                'PNDVI',
                'NBR',
                'NBR2',
                'NDWI_2',
                'SCI',
                'CVI',
                'CCCI',
                'bs','pv','npv'
               ]
    
    importances = rf.feature_importances_
    std = np.std([tree.feature_importances_ for tree in rf.estimators_],
                axis=0)
    indices = np.argsort(importances)[::-1]

    plt.figure()
    plt.title("Feature importances")
    plt.bar(range(X.shape[1]),
            importances[indices],
            color="r",
            yerr=std[indices],
            align="center"
           )
    plt.xticks(range(X.shape[1]), indices)
    plt.xlim([-1, X.shape[1]])
    plt.show()
    
    print("Feature ranking:")

    for f in range(X.shape[1]):
        print("%d. Feature %d (%s): %f" % (f+1, indices[f], FEATURES[indices[f]], importances[indices[f]]))<jupyter_output><empty_output><jupyter_text>Print ScoresWe will display the mean accuracy and F1 scores for the classifier as tested against our test sets.<jupyter_code>def print_scores(rf, feature_set, y_pred):
    """Prints the mean accuracy and F1 scores
    
    Args:
        rf (RandomForestClassifier): The fitted classifier.
        feature_set (dict): The dictionary with the testing data. Should have X_test and y_test keys.
        y_pred: The results of a prediction with y_test data.
    """
    X_test = feature_set['X_test']
    y_test = feature_set['y_test']
    
    mean_accuracy = rf.score(X_test, y_test)

    print("The mean accuracy of the model: %f" % mean_accuracy)
    
    labels=["Forest","Prairie","Summercrops","Naturalgrassland", "Misc"]
    labels.sort()


    f1_result = f1_score(y_test, y_pred, labels=labels, average=None)
    print("The F1 scores of the model:")
    for f in range(len(labels)):
        print('\t%s: %f' % (labels[f], f1_result[f]))<jupyter_output><empty_output><jupyter_text>Plot a Confusion MatrixWe will plot a confusion matrix which will give us an intuitive visualization of the classifier's performance.<jupyter_code>class ConfusionMatrix:
    """A ConfusionMatrix object for displaying confusion matrices.
    
    Attributes:
        cnf_matrix: A calculated confusion matrix from the input variables.
        classes: The list of classes to plot.
    """
    def __init__(self, y_test, y_pred, classes):
        """Initializes the object with the given parameters.
        
        Args:
            y_test: The true test labels.
            y_pred: The predicted test labels.
            classes: A list of the unique labels in the data.
        """
        self.cnf_matrix = self.compute_confusion_matrix(y_test, y_pred)
        classes.sort()
        self.classes = classes
    
    def plot_confusion_matrix(self, normalize=False, title='Confusion matrix', cmap=plt.cm.Blues):
        """Plots a confusion matrix.
        
        Args:
            normalize: Indicator for whether the matrix should be normalized or not.
            title: The title for the confusion matrix.
            cmap: The cmap that the confusion matrix will use for coloring.
            
        Returns:
            The plotted confusion matrix.
        
        Citiation:
        https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html
        ---------
        This function prints and plots the confusion matrix.
        Normalization can be applied by setting `normalize=True`.
        """
        cm = self.cnf_matrix
        classes = self.classes
        
        if normalize:
            cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
            print("Normalized confusion matrix")
        else:
            print('Confusion matrix, without normalization')

        print(cm)

        plt.imshow(cm, interpolation='nearest', cmap=cmap)
        plt.title(title)
        plt.colorbar()
        tick_marks = np.arange(len(classes))
        plt.xticks(tick_marks, classes, rotation=45)
        plt.yticks(tick_marks, classes)

        fmt = '.2f' if normalize else 'd'
        thresh = cm.max() / 2.
        for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
            plt.text(j, i, format(cm[i, j], fmt),
                     horizontalalignment="center",
                     color="white" if cm[i, j] > thresh else "black")

        plt.ylabel('True label')
        plt.xlabel('Predicted label')
        plt.tight_layout()
        
        return plt.show()

    def compute_confusion_matrix(self, y_test, y_pred):
        """Computes the confusion matrix.
        
        Args:
            y_test: The true test labels.
            y_pred: The predicted test labels.
            
        Returns:
            A computed confusion matrix.
        """
        # Compute confusion matrix
        cnf_matrix = confusion_matrix(y_test, y_pred)
        np.set_printoptions(precision=2)
        return cnf_matrix<jupyter_output><empty_output><jupyter_text>Export ModelA function to export the model.<jupyter_code>def export_rf(model, path=None):
    """Exports a RandomForestClassifier.
    
    Args:
        model (RandomForestClassifier): The RandomForestClassifier to export.
        path (string): An optional path to export the model to.
    """
    if(path is None):
        time = datetime.datetime.now().strftime("%Y%m%d%H_%M")
        path = "./classifiers/models/random_forest_{creation_time}.model".format(creation_time = time)
        
    joblib.dump(rf, path)
    print('Model saved as: %s' % path)<jupyter_output><empty_output><jupyter_text>--- Results Import Data<jupyter_code>feature_set = get_features()
features = feature_set['features']
labels = feature_set['labels']
ndvi_coeff_var = feature_set['features']['NDVI_coeff_var']<jupyter_output><empty_output><jupyter_text>Split Data into Training and Testing Sets<jupyter_code>feature_set = split_features(feature_set)
feature_set<jupyter_output><empty_output><jupyter_text>Grid SearchVery slow (typically takes 1h 20m), and the results are about the same as a randomized search. You can uncomment to use it for yourself.<jupyter_code>#rf = grid_search(feature_set)
#rf<jupyter_output><empty_output><jupyter_text>Random SearchFaster than Grid Search (takes about 2m for 50 iterations)<jupyter_code>rf = random_search(feature_set, 100)
rf<jupyter_output><empty_output><jupyter_text>Show Important Features<jupyter_code>importances_plot(rf, features.values)<jupyter_output><empty_output><jupyter_text>Print ScoresThe mean accuracy show us the fraction of correct predictions.The F1 score shows us the weighted harmonic mean of precision and recall. Precision is $\frac{true\_positives}{true\_positives + false\_positives}$ Recall is $\frac{true\_positives}{true\_positives + false\_negatives}$<jupyter_code>y_pred = rf.predict(feature_set['X_test'])

print_scores(rf, feature_set, y_pred)<jupyter_output>The mean accuracy of the model: 0.698113
The F1 scores of the model:
	Forest: 0.666667
	Misc: 0.500000
	Naturalgrassland: 0.847059
	Prairie: 0.222222
	Summercrops: 0.673684<jupyter_text>Show Confusion Matrix<jupyter_code>classes = ['Forest',
           'Misc',
           'Naturalgrassland',
           'Prairie',
           'Summercrops'
          ]

cm = ConfusionMatrix(feature_set['y_test'], y_pred, classes)

plt.figure(figsize=(10, 8))
cm.plot_confusion_matrix(title='Confusion matrix, without normalization')

plt.figure(figsize=(10, 8))
cm.plot_confusion_matrix(normalize=True, title='Normalized confusion matrix')<jupyter_output>Confusion matrix, without normalization
[[ 2  0  0  0  1]
 [ 1  2  0  0  3]
 [ 0  0 36  0  7]
 [ 0  0  0  2 13]
 [ 0  0  6  1 32]]<jupyter_text>--- Export Model for Later UseWe will export the model as a binary file. This will allow us to use the model we've generated for classification of new data elsewhere.<jupyter_code>export_rf(rf)<jupyter_output>Model saved as: ./classifiers/models/random_forest_2019030521_02.model
{'fraction_non_alphanumeric': 0.09811730629978277, 'fraction_numerical': 0.01947863866763215, 'mean_word_length': 2.6192348008385746, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 63, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '190661', 'license': 'Apache-2.0', 'n_tokens_mistral': 4418, 'n_tokens_neox': 4083, 'n_words': 1106, 'path': 'notebooks/machine_learning/Uruguay_Random_Forest/Random_Forest/3. Model Building and Evaluation.ipynb', 'repo_name': 'AnnetteB-MQ/data_cube_notebooks', 'size': '13810'}