<jupyter_start><jupyter_text>Evaluating the Performance of the ClassifierIn this notebook we are going to evaluate the performance of the K-NN classifier built for the **Iron Ore** dataset. In order to have a clean notebook, some functions are implemented in the file *utils.py* (e.g., plot_learning_curve). We are not going to discuss the implementation aspects of these functions as it is not the scope, but you can explore and read the content of the functions later on.Summary: - [Data Pre-processing](data_preprocessing) - [Building the K-Nearest Neighbors Classifier](models) - [Quantifying the Quality of Predictions](pred)     - [Accuracy](accuracy)     - [Confusion Matrix](confusion)     - [Precision and Recall](precision)     - [F1-score](f1score)     - [Receiver Operating Characteristic (ROC) and Area Under the Curve (AUC)](roc)     - [K-fold Cross-Validation](kfold)     - [Learning Curve](learning)  __All the libraries used in this notebook are Open Source__.  Data Pre-processing<jupyter_code># Standard libraries
import numpy as np  # written in C, is faster and robust library for numerical and matrix operations
import pandas as pd # data manipulation library, it is widely used for data analysis and relies on numpy library.
import matplotlib.pyplot as plt # for plotting
import seaborn as sns # plot nicely =)

from sklearn.model_selection import train_test_split #split arrays or matrices into random train and test subsets
from sklearn.preprocessing import StandardScaler #Standardize features by removing the mean and scaling to unit variance

# Auxiliar functions
from utils import *

# the following to lines will tell to the python kernel to always update the kernel for every utils.py
# modification, without the need of restarting the kernel.
%load_ext autoreload
%autoreload 2

# using the 'inline' backend, your matplotlib graphs will be included in your notebook, next to the code
%matplotlib inline<jupyter_output><empty_output><jupyter_text>For this first example, we will work on the same dataset as the previous notebook, the **Iron Ore** dataset.<jupyter_code># reading dataset
df = pd.read_csv('../data/iron_ore_study.csv')

# Splits from oscar Fe>60%, SiO2<9, Al2O3<2, P<0.08
split_points = [
    ('FE', 60, [False, True]),
    ('SIO2', 9, [True, False]),
    ('AL2O3', 2, [True, False]),
    ('P', 0.08, [True, False]),  
]

# It's ore if everything is True
df['is_ore'] = np.vstack([
    pd.cut(df[elem], bins=[0, split, 100], labels=is_ore)
    for elem, split, is_ore in split_points
]).sum(axis=0) == 4

y = df.is_ore # set the variable 'y' to store the labels
# removing is_ore from the dataframe 
list = ['is_ore']
df = df.drop(list,axis = 1 )

# split data train 70% and test 30%. You can try other splits here.
x_train, x_test, y_train, y_test = train_test_split(df, y, test_size=0.3, 
                                                    random_state=42)

y_train = y_train.values # converting to numpy array
y_test = y_test.values # converting to numpy array

# normalising the data
scaler = StandardScaler()
scaler.fit(x_train) # not considering the label is_ore
x_train_scaled = scaler.transform(x_train)

x_test_scaled = scaler.transform(x_test)<jupyter_output><empty_output><jupyter_text>Building the K-Nearest Neighbors Classifier<jupyter_code>from sklearn.neighbors import KNeighborsClassifier #Classifier implementing the k-nearest neighbors vote.

clf_knn = KNeighborsClassifier(n_neighbors=3)   

clf_knn.fit(x_train_scaled, y_train)<jupyter_output><empty_output><jupyter_text>Quantifying the Quality of PredictionsThe performance assessment of the **classifiers** is extremely important in practice, as this provide insights of how the classifier performs with new data, in which me measure the **generalisation error**.Summary: - [Accuracy](accuracy) - [Confusion Matrix](confusion) - [Precision and Recall](precision) - [F1-score](f1score) - [Receiver Operating Characteristic (ROC) and Area Under the Curve (AUC)](roc) - [K-fold Cross-Validation](kfold) - [Learning Curve](learning)  AccuracyComputes the accuracy of the classifier by  using the following equation:$$accuracy = \frac{1}{N}\sum_{i=0}^N{1*(\hat{y}==y)}$$where $y$ is the true label, $\hat{y}$ the predicted label and $N$ is the number of samples.<jupyter_code>from sklearn.metrics import accuracy_score

# we are using prediction computed previsously
prediction = clf_knn.predict(x_test_scaled)

accuracy_knn = accuracy_score(y_test, prediction)

print('Accuracy: ', accuracy_knn)<jupyter_output>Accuracy:  0.9742857142857143<jupyter_text>Nice! We just got 97.4% of **accuracy**. Can we assume that we are done? Let's use other tools to quantify the quality of predictions for our classifier.  Confusion MatrixThe confusion matrix is a tool/technique for summarising the performance of the classifier. We can have better insights about when the classifier is getting right and what are the types of errors it is making. It can be very useful for a further improvement of our models.In the image above, we have:  - **True Positive (TP):** correctly predicted event values. - **False Positive (FP):** incorrectly predicted event values. - **True Negative (TN):** correctly predicted no-event values. - **False Negative (FN):** incorrectly predicted no-event values.<jupyter_code>#  confusion_matrix: computes confusion matrix to evaluate the accuracy of a classification
from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, prediction)
sns.heatmap(cm, annot=True,fmt="d")<jupyter_output><empty_output><jupyter_text>Precision-RecallThe confusion matrix give us a lot of information, but sometimes we may need a better metric in order to evaluate the classifier accuracy. Precision and recall scores are two metrics naturally provided by confusion matrix evaluation.Precision is given by the equation:$$precision = \frac{TP}{TP + FP}$$where $TP$ is the number of True Positives and $FP$ is the number of False Positives.Recall is given by the equation:$$recall  = \frac{TP}{TP + FN}$$where $TP$ is the number of True Positives and $FN$ is the number of False Negatives.<jupyter_code># Libraries:
#   precision_score: computes precision score
#   recall_score: computes recall score
from sklearn.metrics import precision_score, recall_score

# we are using prediction computed previsously
precision_knn = precision_score(y_test, prediction)
recall_knn = recall_score(y_test, prediction)

print('Precision: ', precision_knn)
print('Recall: ', recall_knn)<jupyter_output>Precision:  0.9733727810650887
Recall:  0.9865067466266867<jupyter_text>F1-scoreAlso known as **F-Measure**, can be interpreted as a weighted average of the precision and recall. The formula for F1-score is:$$f_1 = 2 * \frac{precision * recall}{precision+recall}$$<jupyter_code>from sklearn.metrics import f1_score # computes the f1 score

# we are using prediction computed previsously
f1_score_knn = f1_score(y_test, prediction)
print('F1-score: ', f1_score_knn)<jupyter_output>F1-score:  0.9798957557706627<jupyter_text>Receiver Operating Characteristic (ROC) and Area Under the Curve (AUC) Receiver Operating Characteristic (ROC) ROC curve plot the *true positive rate-TPR* (a.k.a. *recall score*) against the *false negative rate-FPR*. FPR is the ratio of negative instances that are incorrectly classifier as positive. Area Under the Curve (AUC)<jupyter_code># Libraries:
#  roc_curve: computes the receiver operating characteristic curve
#  roc_auc_score: computes Area Under the Receiver Operating Characteristic Curve score
from sklearn.metrics import roc_curve, roc_auc_score

# computing and plotting the ROC curve
fpr, tpr, _ = roc_curve(y_test, prediction)

plot_roc_curve(fpr=fpr, tpr=tpr)

# computing the AUC (Area Under the Curve)
auc_knn = roc_auc_score(y_test, prediction)
print('AUC: ', auc_knn)<jupyter_output><empty_output><jupyter_text>K-fold Cross-Validation<jupyter_code># We are using the library StratifiedKFold for this task.
#   The StratifiedKFold performs stratified sampling to produce folds that contain a representative ratio of each 
#   class. At each iteraction the code create a clone of the classifier, train that clone on the training 
#   folds, and makes prediction on the test fold. 
from sklearn.model_selection import StratifiedKFold
from sklearn.base import clone

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

auc_scores = []
f1_scores = []
for train, test in cv.split(x_train_scaled, y_train):
    clone_clf = clone(clf_knn)
    
    # splitting the training set
    x_train_folds = x_train_scaled[train]
    y_train_folds = y_train[train]
    x_test_folds = x_train_scaled[test]
    y_test_folds = y_train[test]
    
    # building the classifier
    clone_clf.fit(x_train_folds, y_train_folds)
    y_pred = clone_clf.predict(x_test_folds)
    
    # computing the auc score
    auc_scores.append(roc_auc_score(y_test_folds, y_pred))
    
    # computing the f1-score
    f1_scores.append(f1_score(y_test_folds, y_pred))
    

fold = 1
print('Fold\tAUC\tF1-score')
for auc, f1 in zip(auc_scores, f1_scores):
    print('{}\t{:.3f}\t{:.3f}'.format(fold, auc, f1))
    fold += 1

print()
print('Mean AUC: {:.3f}'.format(np.array(auc_scores).mean()))
print('Mean F1-score: {:.3f}'.format(np.array(f1_scores).mean()))<jupyter_output>Fold	AUC	F1-score
1	0.959	0.973
2	0.964	0.973
3	0.973	0.979
4	0.959	0.975
5	0.953	0.969

Mean AUC: 0.962
Mean F1-score: 0.974<jupyter_text>Learning Curve<jupyter_code>from sklearn.model_selection import StratifiedKFold

# SVC is more expensive so we do a lower number of CV iterations:
cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)

plot_learning_curve(clf_knn, title='K-NN Learning Curve', 
                    X=x_train_scaled, 
                    y=y_train, 
                    ylim=(0.85, 1.01), 
                    cv=cv, 
                    n_jobs=5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09100987704091917, 'fraction_numerical': 0.02126587381576295, 'mean_word_length': 4.850825471698113, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '500845', 'license': 'MIT', 'n_tokens_mistral': 3325, 'n_tokens_neox': 3076, 'n_words': 1055, 'path': 'notebooks/.ipynb_checkpoints/pm1-performance-evaluation-checkpoint.ipynb', 'repo_name': 'core-skills/06-machine-learning', 'size': '9922'}