<jupyter_start><jupyter_text>Classification<jupyter_code>%load ../helpers/header.py
# # load ENV path to project from .profile 
# import os, sys
# PROJECT_ROOT_DIR=os.environ.get('ML_PATH')
# sys.path.append(os.path.join(PROJECT_ROOT_DIR, "helpers")) # add helper modules to path

# # MPL 
# import MPL_header #load common MPL imports (from helpers)
# import matplotlib.pyplot as plt
# %matplotlib inline 
# # %matplotlib widget 

# # NP, constant seed, PD 
# import numpy as np
# np.random.seed(12345)
# import pandas as pd

# # Where to save the figures
# SUB_DIR = "classification"
# IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, "fig", SUB_DIR)
# # IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, "fig")
# os.makedirs(IMAGES_PATH, exist_ok=True)
from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784')
mnist.keys()
X, y = mnist["data"], mnist["target"]
print("70k digits, each 28x28=784 pixels", X.shape, y.shape, "each pixel goes from 0(White)-255(Black), e.g. X[12][323]=", X[12][323])<jupyter_output>70k digits, each 28x28=784 pixels (70000, 784) (70000,) each pixel goes from 0(White)-255(Black), e.g. X[12][323]= 249.0<jupyter_text>Plot a digit<jupyter_code>some_digit = X[0]
some_digit_image = some_digit.reshape(28, 28)
plt.imshow(some_digit_image, cmap="binary");
plt.axis("off");
print(y[0], "looks like this")
print("Object/string....,", y.dtype)
y = y.astype(np.uint8)
print("Casting to int....,", y.dtype)
X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]<jupyter_output><empty_output><jupyter_text>Already shuffled - good! Ensures: a) folds have equal split b) no repeatability in training. Shuffling is not good for e.g. time-series data (e.g. stcoks)   Binary Classifier: only try to say '5' or 'not 5'<jupyter_code># T/F 
y_train_5 = (y_train == 5)
y_test_5 = (y_test == 5)
from sklearn.linear_model import SGDClassifier
sgd_clf = SGDClassifier(random_state=1234)
sgd_clf.fit(X_train, y_train_5)
sgd_clf.predict([some_digit])<jupyter_output><empty_output><jupyter_text>Performance Measures<jupyter_code>from sklearn.model_selection import cross_val_score
cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring="accuracy")
from sklearn.base import BaseEstimator

class Never5Classifier(BaseEstimator):
    def fit(self, X, y=None):
        return self
    def predict(self, X):
        return np.zeros((len(X), 1), dtype=bool)
never_5_clf = Never5Classifier()
print("Accuracy is not the best metric for classifiers...(generally)")
cross_val_score(never_5_clf, X_train, y_train_5, cv=3, scoring="accuracy")<jupyter_output>Accuracy is not the best metric for classifiers...(generally)<jupyter_text>Confusion Matrix<jupyter_code>from sklearn.model_selection import cross_val_predict
#instead of scores, returns the actual predictions (clean, predictions are on the validation fold)
y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)
from sklearn.metrics import confusion_matrix
# (true, pred)
confusion_matrix(y_train_5, y_train_pred)<jupyter_output><empty_output><jupyter_text>Row = actual class, column= predicted classCM_00 = TN, CM_11 =TP, CM_10 = FN, CM_01 = FP  Accuracy = $\frac{TP+TN}{TP+TN+FP+FN}$Precision = $\frac{TP}{TP+FP}$  (**penalty for mis-firing (over-classifing)**, "how well did it trigger?"/percentage of your results which are relevant)  Really, need a second metric (otherwise e.g. 1/1=100%, not constrained).Recall (sensitivity or true positive rate (TPR)) = $\frac{\rm{TP}}{\rm{TP+FN}}$ (**penalty for "missed out cases"/ (how well did it trigger given truth) (under-classifiing)**/percentage of total relevant results correctly classified by your algorithm)<jupyter_code>from sklearn.metrics import precision_score, recall_score
precision_score(y_train_5, y_train_pred) # == 4296 / (4296 + 1152)
recall_score(y_train_5, y_train_pred) # == 4296/ (4296+1125)<jupyter_output><empty_output><jupyter_text>Harmonic mean gives much more weight to low values. As a result, the classifier will only get a high F1 score if both recall and precision are high.$$F_1 = \frac{2}{\frac{1}{\rm{precision}}+\frac{1}{\rm{recall}}}=2\times\frac{\rm{precision}\times\rm{recall}}{\rm{precision}+\rm{recall}}=\frac{TP}{TP+\frac{FN+FP}{2}}$$<jupyter_code>from sklearn.metrics import f1_score
f1_score(y_train_5, y_train_pred)<jupyter_output><empty_output><jupyter_text>The F1 score favors classifiers that have similar precision and recall. This is not always what you want! (e.g. if you trained a classifier to detect videos that are safe for kids, you would probably prefer a classifier that rejects many good videos (low recall) but keeps only safe ones (high precision))<jupyter_code>y_scores = sgd_clf.decision_function([some_digit]); y_scores
threshold = 0
y_some_digit_pred = (y_scores > threshold); y_some_digit_pred
threshold = 8000
y_some_digit_pred = (y_scores > threshold)
y_some_digit_pred
#return scores instead of predictions
y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, method="decision_function")
# using scores -> prec/reacall for all THRs
from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)
def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):
    plt.plot(thresholds, precisions[:-1], "b--", label="Precision")
    plt.plot(thresholds, recalls[:-1], "g-", label="Recall")
    plt.legend()
    plt.xlabel("Threshold")
    plt.axis([-50000, 50000, -0.001, 1.001])

plot_precision_recall_vs_threshold(precisions, recalls, thresholds)
recall_90_precision = recalls[np.argmax(precisions >= 0.9)]
threshold_90_precision = thresholds[np.argmax(precisions >= 0.9)]
plt.plot([threshold_90_precision, threshold_90_precision], [0., 0.9], "r:");
plt.plot([-50000, threshold_90_precision], [0.9, 0.9], "r:")
plt.plot([-50000, threshold_90_precision], [recall_90_precision, recall_90_precision], "r:", label="Threshold at 90%\n precision: {:.2f}, \nrecall={:.2f}".format(threshold_90_precision, recall_90_precision))
plt.plot([threshold_90_precision], [0.9], "ro")                                            
plt.plot([threshold_90_precision], [recall_90_precision], "ro")                            
plt.legend(loc="center right")
# from misc import save_fig
# save_fig("precision_recall_vs_threshold_plot")  
def plot_precision_vs_recall(precisions, recalls):
    plt.plot(precisions[:-1], recalls[:-1])
    plt.legend()
    plt.xlabel("Recall")
    plt.ylabel("Precision")
    plt.axis([0.001, 1.001, 0.001, 1.001])

plot_precision_vs_recall(precisions, recalls)
plt.plot([recall_90_precision, recall_90_precision], [0.0, 0.9], "r--");
plt.plot([0.0, recall_90_precision], [0.9, 0.9], "r--");
y_train_pred_90 = (y_scores >= threshold_90_precision)
precision_score(y_train_5, y_train_pred_90)
recall_score(y_train_5, y_train_pred_90)<jupyter_output><empty_output><jupyter_text>If someone says, “Let’s reach 99% precision,” you should ask, “At what recall?”<jupyter_code>### ROC: TPR (recall) vs FPR (negative inc. classified as post) ( recall vs. 1- spec., where specificity=TNR)
from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_train_5, y_scores)
def plot_roc_curve(fpr, tpr, label=None): 
    plt.plot(fpr, tpr, linewidth=2, label=label) 
    plt.plot([0, 1], [0, 1], 'k--') # Dashed diagonal
    plt.xlabel('FPR')
    plt.ylabel('Recall (TPR)')
    plt.grid(linewidth=1)
    plt.legend(fontsize=14)

plot_roc_curve(fpr, tpr, label="SGD") 
plt.show()
 # perfect AUC for ROC is 1
from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_5, y_scores)<jupyter_output><empty_output><jupyter_text>Since the ROC curve is so similar to the precision/recall (PR) curve, you may wonder how to decide which one to use. As a rule of thumb, you should prefer the PR curve whenever the positive class is rare or when you care more about the false positives than the false negatives. Otherwise, use the ROC curve. For example, looking at the previous ROC curve (and the ROC AUC score), you may think that the classifier is really good. But this is mostly because there are few positives (5s) compared to the negatives (non-5s). In contrast, the PR curve makes it clear that the classifier has room for improvement (the curve could be closer to the top- left corner).<jupyter_code>from sklearn.ensemble import RandomForestClassifier
forest_clf = RandomForestClassifier(random_state=123)
y_probas_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3, method="predict_proba")
print("Binary prob class scores:")
y_probas_forest
y_scores_forest = y_probas_forest[:, 1] # score = proba of positive class
fpr_forest, tpr_forest, thresholds_forest = roc_curve(y_train_5,y_scores_forest)
plt.plot(fpr, tpr, "b:", label="SGD") 
plot_roc_curve(fpr_forest, tpr_forest, "Random Forest")
plt.legend(loc="lower right")
plt.show()
print("Random Forest performs better", roc_auc_score(y_train_5, y_scores_forest))
y_train_pred_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3)
forest_ps= precision_score(y_train_5, y_train_pred_forest)
forest_rs = recall_score(y_train_5, y_train_pred_forest)
print("Random Forest: precision={0:.2f}, recall={1:.2f}".format(forest_ps, forest_rs))<jupyter_output>Random Forest: precision=0.99, recall=0.87<jupyter_text>Multiclass Classification  OvR (e.g. "1"-detecor, so 10 for MNIST) or OvO ("0s vs 1s", "0s vs 2s"... n*(n-1)/2 = 45, but only need to train on subset of data). sickit will chose one depending on the algorithm used (SVM is OvR for performance)<jupyter_code>from sklearn.multiclass import OneVsRestClassifier
ovr_clf = OneVsRestClassifier(SVC(gamma="auto", random_state=42))
ovr_clf.fit(X_train[:1000], y_train[:1000])
ovr_clf.predict([some_digit])
some_digit_scores = svm_clf.decision_function([some_digit])
some_digit_scores
np.argmax(some_digit_scores)
svm_clf.classes_
svm_clf.classes_[5]
from sklearn.multiclass import OneVsRestClassifier
ovr_clf = OneVsRestClassifier(SVC(gamma="auto", random_state=42))
ovr_clf.fit(X_train[:1000], y_train[:1000])
ovr_clf.predict([some_digit])
 len(ovr_clf.estimators_)
sgd_clf.fit(X_train, y_train)
sgd_clf.predict([some_digit])
sgd_clf.decision_function([some_digit])<jupyter_output><empty_output><jupyter_text>Evaluating the calssifier usin CV<jupyter_code>cross_val_score(sgd_clf, X_train, y_train, cv=3, scoring="accuracy")
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
cross_val_score(sgd_clf, X_train_scaled, y_train, cv=3, scoring="accuracy")<jupyter_output><empty_output><jupyter_text>Error Analysis<jupyter_code>y_train_pred = cross_val_predict(sgd_clf, X_train_scaled, y_train, cv=3)
 conf_mx = confusion_matrix(y_train, y_train_pred)
 conf_mx
plt.matshow(conf_mx, cmap=plt.cm.gray)
plt.show()<jupyter_output><empty_output><jupyter_text>The 5s look slightly darker than the other digits, which could mean that there are fewer images of 5s in the dataset or that the classifier does not perform as well on 5s as on other digits.<jupyter_code># 1) normalise the confusion matrix
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = conf_mx / row_sums
# Fill the diagonal with zeros to keep only the errors, and plot the result
np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
plt.show()<jupyter_output><empty_output><jupyter_text>"Row" = actual class, Column=predicted classs. From above: many images get mis-classified as 8s. But 8s themselves get classified well. Aim: reduce false-postive 8s. Solutions: 1) More training data taht look like 8s, but are not. 2) Engineer new features: e.g. "closed-loops" (8 has 2, 6 has 1 etc.)<jupyter_code>cl_a, cl_b = 3, 5
X_aa = X_train[(y_train == cl_a) & (y_train_pred == cl_a)] 
X_ab = X_train[(y_train == cl_a) & (y_train_pred == cl_b)] 
X_ba = X_train[(y_train == cl_b) & (y_train_pred == cl_a)] 
X_bb = X_train[(y_train == cl_b) & (y_train_pred == cl_b)]
plt.figure(figsize=(8,8))
plt.subplot(221); plot_digits(X_aa[:25], images_per_row=5) 
plt.subplot(222); plot_digits(X_ab[:25], images_per_row=5) 
plt.subplot(223); plot_digits(X_ba[:25], images_per_row=5) 
plt.subplot(224); plot_digits(X_bb[:25], images_per_row=5) 
plt.show()<jupyter_output><empty_output><jupyter_text>Multilabel Classification<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
y_train_large = (y_train >= 7)
y_train_odd = (y_train % 2 == 1)
y_multilabel = np.c_[y_train_large, y_train_odd]
knn_clf = KNeighborsClassifier() 
knn_clf.fit(X_train, y_multilabel)
#The digit 5 is indeed not large (False) and odd (True). 
knn_clf.predict([some_digit])<jupyter_output><empty_output><jupyter_text>computes the average F1 score across all labels<jupyter_code>y_train_knn_pred = cross_val_predict(knn_clf, X_train, y_multilabel, cv=3)
f1_score(y_multilabel, y_train_knn_pred, average="macro")<jupyter_output><empty_output><jupyter_text>Multioutput Classification<jupyter_code>noise = np.random.randint(0, 100, (len(X_train), 784)) 
X_train_mod = X_train + noise
noise = np.random.randint(0, 100, (len(X_test), 784)) 
X_test_mod = X_test + noise
y_train_mod = X_train y_test_mod = X_test
knn_clf.fit(X_train_mod, y_train_mod)
clean_digit = knn_clf.predict([X_test_mod[some_index]]) 
plot_digit(clean_digit)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12494371904547501, 'fraction_numerical': 0.03234278853369353, 'mean_word_length': 6.624141876430206, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '317785', 'license': 'MIT', 'n_tokens_mistral': 5315, 'n_tokens_neox': 4841, 'n_words': 1143, 'path': 'JupyterNB/Classification.ipynb', 'repo_name': 'glukicov/ML_GPU', 'size': '13326'}