<jupyter_start><jupyter_text>Introduction to Machine Learning with scikit-learn Lab 5: Model evaluation and selectionIn this lab, we will apply a few model evaluation metrics we've seen in the lecture.<jupyter_code>%matplotlib inline
import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)<jupyter_output><empty_output><jupyter_text>As always, we need to start with some data.Let's first generate a set of outputs $y$ and predicted outputs $\hat{y}$ to illustrate a few typical cases.<jupyter_code>from sklearn.metrics import confusion_matrix
y_true = [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0]
y_pred = [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0]
cnf_matrix = confusion_matrix(y_true, y_pred)
print(cnf_matrix)<jupyter_output>[[12  1]
 [ 0  2]]<jupyter_text>Now let's define a function that will display the confusion matrix. The following is inspired from [this example]( http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html).<jupyter_code>import itertools
import numpy as np
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix

def plot_confusion_matrix(cm, classes,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes)
    plt.yticks(tick_marks, classes)

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
plt.figure()
plot_confusion_matrix(cnf_matrix, ['0', '1'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10557986870897156, 'fraction_numerical': 0.020787746170678335, 'mean_word_length': 4.5932721712538225, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '24650', 'license': 'MIT', 'n_tokens_mistral': 633, 'n_tokens_neox': 591, 'n_words': 175, 'path': 'it_step/intro_ml/lab_5/model_selection_validation.ipynb', 'repo_name': 'azubiolo/lectures', 'size': '1828'}