<jupyter_start><jupyter_text>Machine Learning Metrics Determining Model AccuracyIt's very common to train a variety of models, apply each to held out sample andscore the results. Sometimes, a third hold set is used to test a model on_completely_ new data. Typically, these 2 hold out sets are called the_validation_ and _test_ sets respectively. In order to evaluate a model, asuitable metric for the dataset needs to be selected.Datasets with rare occurring labellings can produce misleading model performanceif a less nuanced metric is chosen to measure it. The following are some definitions to help us better understandthese metrics:- True Positive (TP): True positive represents the value of correct predictions of positives out of actual positive cases.- False Positive (FP): False positive represents the value of incorrect positive predictions.- True Negative (TN): True negative represents the value of correct predictions of negatives out of actual negative cases.- False Negative (FN): False negative represents the value of incorrect negative predictions. AccuracySimple accuracy is the measure of how many discrete labellings by a classifier(or regressor) were correct.*Accuracy Score = (TP + TN) / (TP + FN + TN + FP)*<jupyter_code>import numpy as np
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score

# Simple synthetic data
training_points = [[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]]
training_labels = [1, 1, 1, 2, 2, 2]
X = np.array(training_points)
Y = np.array(training_labels)

# Gaussian Naive Bayes classifier
gnb = GaussianNB()
gnb.fit(X, Y)

# Classify test data with the classifier
test_points = [[1, 1], [2, 2], [3, 3], [4, 3]]
test_labels = [2, 2, 2, 1]
predicts = gnb.predict(test_points)

count = len([True for idx, label in enumerate(test_labels) if label == predicts[idx]])
print("Accuracy Rate (manually calculated): %f" % (float(count) / len(test_labels)))
print("Accuracy Rate (via accuracy_score()): %f" % accuracy_score(test_labels, predicts))<jupyter_output>Accuracy Rate (manually calculated): 0.750000
Accuracy Rate (via accuracy_score()): 0.750000<jupyter_text>What about accuracy and cross validation?<jupyter_code>from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score

kf = KFold(n_splits=5)
result = cross_val_score(gnb , X, Y, cv = kf)
print(result)
print("Avg accuracy: {}".format(result.mean()))<jupyter_output>[0. 0. 1. 1. 0.]
Avg accuracy: 0.4<jupyter_text>Recall, Precision and F-Measure, Oh My!To better understand your model's effectiveness use the Recall, Precision andF-Measure metrics._Recall_ is the measure of how many correct labellings your model predicted. Itis defined as: *Recall Score = TP / (FN + TP)*_Precision_ is the measure of how many incorrect labellings your model predicted.*Precision Score = TP / (FP + TP)*_F-measure_ is something like an average of the two scores.*F1 Score = 2 * Precision Score * Recall Score / (Precision Score + Recall Score)*<jupyter_code>from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score
from sklearn import datasets
from sklearn.model_selection import train_test_split

bc = datasets.load_breast_cancer()
X = bc.data
Y = bc.target

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.30, random_state=1, stratify=Y)

sc = StandardScaler()
sc.fit(X_train)
X_train_std = sc.transform(X_train)
X_test_std = sc.transform(X_test)

# train the support vector machine
svc = SVC(kernel='linear', C=10.0, random_state=1)
svc.fit(X_train, y_train)

y_pred = svc.predict(X_test)

print('Recall: %.3f' % recall_score(y_test, y_pred))
print('Precision: %.3f' % precision_score(y_test, y_pred))
print('Accuracy: %.3f' % accuracy_score(y_test, y_pred))
print('F1 Score: %.3f' % f1_score(y_test, y_pred))<jupyter_output>Recall: 0.972
Precision: 0.972
Accuracy: 0.965
F1 Score: 0.972<jupyter_text>Confusion MatrixA confusion matrix is a standard way to graph TP, TN, FP, FN is a 2x2 matrix,resulting in a nice quadrant display of the results.<jupyter_code>from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt

conf_matrix = confusion_matrix(y_true=y_test, y_pred=y_pred)

fig, ax = plt.subplots(figsize=(5, 5))
ax.matshow(conf_matrix, cmap=plt.cm.Oranges, alpha=0.3)
for i in range(conf_matrix.shape[0]):
    for j in range(conf_matrix.shape[1]):
        ax.text(x=j, y=i,s=conf_matrix[i, j], va='center', ha='center', size='xx-large')

plt.xlabel('Predicted Labels', fontsize=18)
plt.ylabel('Gold Labels', fontsize=18)
plt.title('The Confusion Matrix', fontsize=18)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10467329245083527, 'fraction_numerical': 0.021357580883907802, 'mean_word_length': 5.815561959654179, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '479472', 'license': 'MIT', 'n_tokens_mistral': 1620, 'n_tokens_neox': 1506, 'n_words': 527, 'path': 'Metrics.ipynb', 'repo_name': 'nathan-gilbert/ai-ml-guild-metrics', 'size': '4729'}