<jupyter_start><jupyter_text>Metrics on binary classifications with TensorFlow 2To generate models first run: https://colab.research.google.com/github/DJCordhose/ml-workshop/blob/master/notebooks/tf2/tf-keras-classifier.ipynb<jupyter_code>import matplotlib.pyplot as plt
# plt.xkcd()
# plt.style.use('ggplot')
%matplotlib inline

import matplotlib as mpl
mpl.rcParams['figure.figsize'] = (20, 8)
#@title Configure our example { display-mode: "form", run: "auto" }

# https://colab.research.google.com/notebooks/forms.ipynb

n = 100 #@param {type:"slider", min:1, max:1000, step:1}
m = -1 #@param {type:"slider", min:-10, max:10, step: 0.1}
b = 1 #@param {type:"slider", min:-10, max:10, step: 0.1}

noise_level = 0.1 #@param {type:"slider", min:0.0, max:1.0, step:0.1}

dim_1_label = 'x1' #@param {type:"string"}
dim_2_label = 'x2' #@param {type:"string"}

import numpy as np

# all points
X = np.random.uniform(0, 1, (n, 2))

# below or above line determines which category they belong to (plus noise)
noise = np.random.normal(0, noise_level, n)
y = X[:, 1] > m*X[:, 0]+b + noise
y = y.astype(int)

from matplotlib.colors import ListedColormap

plt.xlabel(dim_1_label)
plt.ylabel(dim_2_label)
plt.title('Categories expressed as colors')

plt.scatter(X[:,0], X[:,1], c=y, cmap=ListedColormap(['#FF6666', '#6666FF']), marker='o', edgecolors='k');<jupyter_output><empty_output><jupyter_text>Load our two competing models<jupyter_code>try:
  # %tensorflow_version only exists in Colab.
  %tensorflow_version 2.x
except Exception:
  pass

import tensorflow as tf
tf.__version__
x = tf.constant(X, dtype='float32')
y_true = tf.constant(y, dtype='float32')
x.shape, y.shape
!curl -O https://raw.githubusercontent.com/DJCordhose/ml-workshop/master/notebooks/tf2/linear-mse.h5
!curl -O https://raw.githubusercontent.com/DJCordhose/ml-workshop/master/notebooks/tf2/sigmoid-crossentropy.h5
!ls -l
model_linear_mse = tf.keras.models.load_model('linear-mse.h5')

y_pred_model_linear_mse = model_linear_mse.predict(x)

plt.hist(y_pred_model_linear_mse, bins=n)
plt.title('Distribution of predictions for linear model using MSE');
model_sigmoid_crossentropy = tf.keras.models.load_model('sigmoid-crossentropy.h5')

y_pred_model_sigmoid_crossentropy = model_sigmoid_crossentropy.predict(x)

plt.hist(y_pred_model_sigmoid_crossentropy, bins=n)
plt.title('Distribution of predictions for sigmoid model using crossentropy');<jupyter_output>WARNING:tensorflow:From /tensorflow-2.0.0-rc1/python3.6/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where<jupyter_text>Basic Metrics* https://en.wikipedia.org/wiki/Precision_and_recall* https://en.wikipedia.org/wiki/F1_score  Most typical metrics* _Precision_: ability not to label a sample as positive (1) that is negative (0)* _Recall_: ability to find all the positive (1) samples* _F1-Score_  * harmonic mean of the precision and recall  * harmonic mean is a good mean for ratios  * best value at 1 and worstat 0* _Support_: number of occurrences of samples for each class<jupyter_code># TensorFlow: https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/metrics
# Scikit-learn: https://scikit-learn.org/stable/modules/model_evaluation.html
# https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
# https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html

from sklearn.metrics import precision_recall_fscore_support, classification_report<jupyter_output><empty_output><jupyter_text>Scores for linear model with MSE, higher is better<jupyter_code>precision, recall, f1_score, support = precision_recall_fscore_support(y_true, (y_pred_model_linear_mse > 0.5).astype(int).reshape(-1))
precision, recall, f1_score, support
print(classification_report(y_true, (y_pred_model_linear_mse > 0.5).astype(int).reshape(-1)))<jupyter_output>precision    recall  f1-score   support

         0.0       0.94      0.98      0.96        51
         1.0       0.98      0.94      0.96        49

    accuracy                           0.96       100
   macro avg       0.96      0.96      0.96       100
weighted avg       0.96      0.96      0.96       100<jupyter_text>Scores for sigmode model with cross entropy, higher is better<jupyter_code>precision, recall, f1_score, support = precision_recall_fscore_support(y_true, (y_pred_model_sigmoid_crossentropy > 0.5).astype(int).reshape(-1))
precision, recall, f1_score, support
print(classification_report(y_true, (y_pred_model_sigmoid_crossentropy > 0.5).astype(int).reshape(-1)))<jupyter_output>precision    recall  f1-score   support

         0.0       1.00      0.94      0.97        51
         1.0       0.94      1.00      0.97        49

    accuracy                           0.97       100
   macro avg       0.97      0.97      0.97       100
weighted avg       0.97      0.97      0.97       100<jupyter_text>How would the scores change if we changed the threshold?  ROC and AUC_graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied__ROC curves typically feature true positive rate on the Y axis, and false positive rate on the X axis. This means that the top left corner of the plot is the “ideal” point - a false positive rate of zero, and a true positive rate of one. This is not very realistic, but it does mean that a larger area under the curve (AUC) is usually better._* https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5* https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc<jupyter_code># https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html
# https://scikit-learn.org/stable/modules/generated/sklearn.metrics.auc.html
# https://scikit-learn.org/stable/modules/model_evaluation.html#roc-metrics

from sklearn.metrics import roc_curve, auc
# roc_curve?
fpr, tpr, thresholds = roc_curve(y_true, y_pred_model_linear_mse)
fpr, tpr, thresholds
roc_auc = auc(fpr, tpr)
roc_auc
# code more or less taken from
# https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html

lw = 2
plt.plot(fpr, tpr, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic, linear model using MSE')
plt.legend(loc="lower right");
fpr, tpr, thresholds = roc_curve(y_true, y_pred_model_sigmoid_crossentropy)
roc_auc = auc(fpr, tpr)

lw = 2
plt.plot(fpr, tpr, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic, sigmoid model using crossentropy')
plt.legend(loc="lower right");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12345335743083553, 'fraction_numerical': 0.03614625330182122, 'mean_word_length': 4.417168674698795, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 27, 'https://': 16, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '511825', 'license': 'MIT', 'n_tokens_mistral': 2719, 'n_tokens_neox': 2535, 'n_words': 593, 'path': 'notebooks/tf2/tf-keras-classifier-metrics.ipynb', 'repo_name': 'kmunve/ml-workshop', 'size': '7193'}