<jupyter_start><jupyter_text>Let's load the MNIST data set for digits:<jupyter_code>from sklearn.datasets import fetch_openml
mnist = fetch_openml("mnist_784", version = 1)
mnist.keys()<jupyter_output><empty_output><jupyter_text>Formatting the data from mnist:<jupyter_code>X, y = mnist["data"], mnist["target"]
print(X.shape)
print(y.shape)
import matplotlib as mpl
import matplotlib.pyplot as plt
 
some_digit = X[0]
some_digit_image = some_digit.reshape(28,28)

plt.imshow(some_digit_image, cmap = "binary")
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>It looks like a 5, but let's check its label:<jupyter_code>y[0]<jupyter_output><empty_output><jupyter_text>Let's convert the labels to integers:<jupyter_code>import numpy as np

y = y.astype(np.uint8)<jupyter_output><empty_output><jupyter_text>Now we split the dataset into training sets and data sets: (Mnist dataset already comes properly formatted for this with 60000/10000)<jupyter_code>X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]<jupyter_output><empty_output><jupyter_text>Training a Binary Classifier<jupyter_code>from sklearn.linear_model import SGDClassifier

y_train_5 = (y_train == 5)
y_test_5 = (y_test == 5)

sgd_clf = SGDClassifier(random_state = 42)
sgd_clf.fit(X_train, y_train_5)

sgd_clf.predict([some_digit])<jupyter_output><empty_output><jupyter_text>Performance Measures Implementing cross validation<jupyter_code>from sklearn.model_selection import StratifiedKFold
from sklearn.base import clone

skfolds = StratifiedKFold(n_splits=3)

for train_index, test_index in skfolds.split(X_train, y_train_5):
    clone_clf = clone(sgd_clf)
    X_train_folds = X_train[train_index]
    y_train_folds = y_train_5[train_index]
    X_test_fold = X_train[test_index]
    y_test_fold = y_train_5[test_index]
    
    clone_clf.fit(X_train_folds, y_train_folds)
    y_pred = clone_clf.predict(X_test_fold)
    n_correct = sum(y_pred == y_test_fold)
    print(n_correct/ len(y_pred))
from sklearn.model_selection import cross_val_score
cross_val_score(sgd_clf, X_train, y_train, cv = 3, scoring = "accuracy")<jupyter_output><empty_output><jupyter_text>Confusion Matrix<jupyter_code>from sklearn.model_selection import cross_val_predict

y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv = 3)

from sklearn.metrics import confusion_matrix

confusion_matrix(y_train_5, y_train_pred)<jupyter_output><empty_output><jupyter_text>The accuracy of positive predictions, called the **precision** of the classifier is given by:precision = $\frac{TP}{TP+FP}$, where TP is the number of True positives and FP is the number of false positivesIn this case, the precision of our classification is $\frac{3530}{687+3530} \approx 0.837 $Precision is typically used along with another metric named **recall**, also called **sentsitivity** or **true positive rate**(TPR) is the retio of positive instances that are correctly detected by the classifier:recall = $\frac{TP}{TP+FN}$, FN is of course the number of false negatives.In this case, the recall is $\frac{3530}{3530+1891} \approx 0.651$  Precision and Recall<jupyter_code>from sklearn.metrics import precision_score, recall_score
precision_score(y_train_5, y_train_pred)

recall_score(y_train_5, y_train_pred)<jupyter_output><empty_output><jupyter_text>It is convenient to to combine precision and recall into a single metric called the **F1 score**. The F1 score is the harmonic mean of precision and recall. (Regular mean treats all values uniformly whereas harmonic mean gives much more weight to the low values). The classifier will only get a high F1 score if both recall and precision are high.F1 = $\frac{2}{\frac{1}{precision} + \frac{1}{recall}} = 2*\frac{precision *recall}{precision+recall} = \frac{TP}{TP+\frac{FN+FP}{2}}$<jupyter_code>from sklearn.metrics import f1_score
f1_score(y_train_5, y_train_pred)<jupyter_output><empty_output><jupyter_text>The f1 score favors classifiers that have similar precision and recall which isn't always ideal. In some contexts, a high precision is wanted and in others a high recall is wanted.Increasing precision reduces recall, and vice-versa. It is called the **precision/recall trade off**.<jupyter_code>y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv = 3, method ="decision_function")
from sklearn.metrics import precision_recall_curve

precisions,recalls, thresholds = precision_recall_curve(y_train_5, y_scores)

def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):
    plt.plot(thresholds, precisions[:-1], "b--", label = "Precision")
    plt.plot(thresholds, recalls[:-1], "g-", label = "Recall")
    [...]

plot_precision_recall_vs_threshold(precisions, recalls, thresholds)
plt.show()    
threshold_90_precision = thresholds[np.argmax(precisions >= 0.90)]
print(threshold_90_precision)
y_train_pred_90 = (y_scores >= threshold_90_precision)
precision_score(y_train_5, y_train_pred_90)

recall_score(y_train_5, y_train_pred_90)<jupyter_output><empty_output><jupyter_text>The ROC Curve<jupyter_code>from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_train_5, y_scores)

def plot_roc_curve(fpr, tpr, label = None):
    plt.plot(fpr,tpr, linewidth = 2, label=label)
    plt.plot([0,1], [0,1], 'k--')
    [...]

plot_roc_curve(fpr,tpr)
plt.show()<jupyter_output><empty_output><jupyter_text>The **ROC curve** is another common tool used with binary classifiers. It is very similar to the precision/recall curve, but instead of plotting precision versus recall, the ROC curve plots the **the true positive rate**(recall) againts the **false positive rate** (FPR). The FPR is the ratio of negative instances that are incorrectly classified as positive. It is equal to 1 - true negative rate (TNR), which is the ratio of negative instances that are correctly classified as negative. TNR is also called **specificity**. ROC curve plots **sensitivity**(recall) versus 1-specificity.  We can also compare classifiers by measuring the area under the curve (AUC). A perfect classifier will have a ROC AUC equal to 1, while a true random one will have a ROC AUC of 0.5.<jupyter_code>from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_5, y_scores)<jupyter_output><empty_output><jupyter_text>choosing PR versus ROC- You should choose the precision/recall curve to evaluate the model when the instance of your positive class is rare.- You should choose the precision/recall curve when you care more about false positives than false negatives.- In any other case, you should use the receiving operator curve.  Random Forest ClassifierLet's compare the SGD classifier to the Random Forest Classifier<jupyter_code>from sklearn.ensemble import RandomForestClassifier

forest_clf = RandomForestClassifier(random_state = 42)
y_probas_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv = 3, method = "predict_proba")

y_scores_forest = y_probas_forest[:,1]
fpr_forest, tpr_forest, thresholds_forest = roc_curve(y_train_5, y_scores_forest)

plt.plot(fpr, tpr, "b:", label = "SGD")
plot_roc_curve(fpr_forest, tpr_forest, "Random Forest")
plt.legend(loc = "lower right")
plt.show()
roc_auc_score(y_train_5, y_scores_forest)<jupyter_output><empty_output><jupyter_text>Multiclass Classification One strategy is to use multiple binary classifiers and choose the one with the highest decision score. This is sometimes called an **OvR**(One versus Rest) strategy or one versus all strategy. Another strategy is to train a classifier for every pair of digits, e.g. 0 versus 1, 0 versus 2, ... 0 versus 9, 1 versus 2, ... 8 versus 9.This is called the **OvO**(one versus one) strategy. If there are N classes, the there are $\frac{N \times (N-1)}{2}$ classifiers to train, or in our case 45 classifiers to train. - Some algorithms scale poorly with size of training set so having many smaller OvO training sets is better- But for most binary classifiers OvR is generally preffered Let's try Support Vector Machine classifier:<jupyter_code>from sklearn.svm import SVC

svm_clf = SVC()
svm_clf.fit(X_train, y_train)
svm_clf.predict([some_digit])
some_digit_scores = svm_clf.decision_function([some_digit])
some_digit_scores
np.argmax(some_digit_scores)
svm_clf.classes_
svm_clf.classes_[5]<jupyter_output><empty_output><jupyter_text>When a classifier is trained it stores the list of target classes in its classes_ attribute. In this case the index and class match, but this won't always be the case. If we want to force Scikit-Learn to use OvO or OvR, we can use OneVsOneClassifier or OneVsRestClassifier classes:<jupyter_code>from sklearn.multiclass import OneVsRestClassifier
over_clf = OneVsRestClassifier(SVC())
over_clf.fit(X_train, y_train)
over_clf.predict([some_digit])
len(over_clf.estimators_)
sgd_clf.fit(X_train, y_train)
sgd_clf.predict([some_digit])
sgd_clf.decision_function([some_digit])
cross_val_score(sgd_clf, X_train, y_train, cv = 3, scoring = "accuracy")

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
cross_val_score(sgd_clf, X_train_scaled, y_train, cv = 3, scoring = "accuracy")<jupyter_output><empty_output><jupyter_text>Error Analysis<jupyter_code>y_train_pred = cross_val_predict(sgd_clf, X_train_scaled, y_train, cv = 3)
conf_mx = confusion_matrix(y_train, y_train_pred)
conf_mx
plt.matshow(conf_mx, cmap = plt.cm.gray)
plt.show()
row_sums = conf_mx.sum(axis = 1, keepdims = True)
norm_conf_mx = conf_mx/row_sums

np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap= plt.cm.gray)
plt.show()<jupyter_output><empty_output><jupyter_text>The rows above represent ther actual classes while the columns represent the predicted classes. We can also directly plot the images of the data that is failing a lot of the tests, to gain a better idea why are model is failing on those particular images.<jupyter_code>def plot_digits(instances, images_per_row=10, **options):
    size = 28
    images_per_row = min(len(instances), images_per_row)
    images = [instance.reshape(size,size) for instance in instances]
    n_rows = (len(instances) - 1) // images_per_row + 1
    row_images = []
    n_empty = n_rows * images_per_row - len(instances)
    images.append(np.zeros((size, size * n_empty)))
    for row in range(n_rows):
        rimages = images[row * images_per_row : (row + 1) * images_per_row]
        row_images.append(np.concatenate(rimages, axis=1))
    image = np.concatenate(row_images, axis=0)
    plt.imshow(image, cmap = mpl.cm.binary, **options)
    plt.axis("off")
    
cl_a,cl_b = 3,5
X_aa = X_train[(y_train == cl_a) & (y_train_pred == cl_a)]
X_ab = X_train[(y_train == cl_a) & (y_train_pred == cl_b)]
X_ba = X_train[(y_train == cl_b) & (y_train_pred == cl_a)]
X_bb = X_train[(y_train == cl_b) & (y_train_pred == cl_b)]
plt.figure(figsize=(8,8))
plt.subplot(221); plot_digits(X_aa[:25], images_per_row = 5)
plt.subplot(222); plot_digits(X_ab[:25], images_per_row = 5)
plt.subplot(223); plot_digits(X_ba[:25], images_per_row = 5)
plt.subplot(224); plot_digits(X_bb[:25], images_per_row = 5)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0994262148108302, 'fraction_numerical': 0.01819974896897974, 'mean_word_length': 6.402123424021235, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 75, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '218013', 'license': 'MIT', 'n_tokens_mistral': 4087, 'n_tokens_neox': 3806, 'n_words': 1054, 'path': 'digitsCNN.ipynb', 'repo_name': 'alexandreLamarre/Digit-recognition-CNN', 'size': '11154'}