<jupyter_start><jupyter_text>Table of Contents* [Intro](Intro)* [Classification](Classification)	* [Accuracy, precision, recall](Accuracy,-precision,-recall)		* [Binary](Binary)		* [Multi-class](Multi-class)* [Probabilities](Probabilities)	* [Log Loss](Log-Loss)* [Regression](Regression)* [Hinge Loss](Hinge-Loss)  Intro Notebook that explores evaluation metrics for different tasks (classification, regression) and scenarios (binary, multi-class, probabilities). See also loss/cost/objective function.<jupyter_code>import numpy as np
import pandas as pd
from sklearn import metrics
import matplotlib.pyplot as plt

import seaborn as sns

sns.set_context("paper")

%matplotlib notebook<jupyter_output><empty_output><jupyter_text>Classification   Accuracy, precision, recall $$ accuracy = \frac{TP+TN}{TP+TN+FP+FN}  $$$$ precision = \frac{TP}{TP+FP}  $$$$ recall = \frac{TP}{TP+FN}  $$  Binary<jupyter_code># dummy data example
y_true = [0, 1, 1, 1, 0, 1, 0, 1, 0, 1]
y_pred = [0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
# accuracy, precision and recall using sklearn
print("Accuracy = {}".format(metrics.accuracy_score(y_true, y_pred)))
print("Precision = {}".format(metrics.precision_score(y_true, y_pred)))
print("Recall = {}".format(metrics.recall_score(y_true, y_pred)))<jupyter_output>Accuracy = 0.5
Precision = 1.0
Recall = 0.16666666666666666<jupyter_text>Multi-class Need an averagin method to compute precision and recall<jupyter_code># dummy data example
y_true = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
y_pred = [0, 2, 1, 3, 3, 3, 6, 7, 9, 9]
# accuracy, precision and recall using sklearn
print("Accuracy = {}".format(metrics.accuracy_score(y_true, y_pred)))
print("Precision = {}".format(metrics.precision_score(y_true, y_pred, average=None)))
print("Recall = {}".format(metrics.recall_score(y_true, y_pred, average=None)))
# Confusion Matrix
metrics.confusion_matrix(y_true, y_pred)<jupyter_output><empty_output><jupyter_text>Probabilities  Log Loss Also cross-entropy loss, a generalization for multiple classes of binary cross-entropy.$$H(p,q)\ =\ -\sum _{i}p_{i}\log q_{i}\ =\ -y\log {\hat  {y}}-(1-y)\log(1-{\hat  {y}})$$<jupyter_code># Log Loss with sklearn
metrics.log_loss(["spam", "ham", "ham", "spam"],  
         [[.1, .9], [.9, .1], [.8, .2], [.35, .65]])
# single binary case
def log_loss(y_true, y_pred):
    return ((-y_true * np.log(y_pred)) - 
             (1-y_true) * (np.log(1-y_pred)))
y_true = 1
x = np.linspace(0.001, 0.9999, 50)
#y = [(1 - y_true) * np.log(1 - y_pred) for y_pred in x]
#y = [(y_true) * np.log(y_pred) for y_pred in x]
y = [log_loss(y_true, y_pred) for y_pred in x]
# plot loss
plt.plot(x, y)
plt.title("Log loss for y_true = {}".format(y_true, margin))
plt.xlabel("y_pred")
plt.ylabel("loss")
plt.show()<jupyter_output><empty_output><jupyter_text>Regression<jupyter_code>y_true = [3, 0, 2, 7]
y_pred = [4, 0, 3, 8]
sns.regplot(x=np.arange(1,len(y_true)+1), y=np.array(y_true), fit_reg =False)
sns.regplot(x=np.arange(1,len(y_true)+1), y=np.array(y_pred), color='r', fit_reg =False)
sns.plt.show()
metrics.mean_absolute_error(y_true, y_pred)
metrics.mean_squared_error(y_true, y_pred)<jupyter_output><empty_output><jupyter_text>Hinge Loss Also max-margin loos (proper of SVMs).$$l(y, \hat{y}) = max(0, 1-(y \cdot \hat{y}))$$As can be observed, $y = \pm1$<jupyter_code>def hinge_loss(y_true, y_pred, margin=1):
    return max(0, margin-(y_true*y_pred))
hinge_loss(1, 1)
# compute loss for different predicted values
y_true = 1
margin = 1
x = np.linspace(-1, 1, 10)
y = [hinge_loss(y_true, y_pred, margin) for y_pred in x]
# plot loss
plt.plot(x, y)
plt.title("Hinge loss for y_true = {}, margin={}".format(y_true, margin))
plt.xlabel("y_pred")
plt.ylabel("loss")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17568291376539905, 'fraction_numerical': 0.030798071772897697, 'mean_word_length': 5.80327868852459, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '135583', 'license': 'Apache-2.0', 'n_tokens_mistral': 1595, 'n_tokens_neox': 1465, 'n_words': 340, 'path': 'machine learning/Evaluation Metrics.ipynb', 'repo_name': 'sugatoray/data-science-learning', 'size': '3734'}