<jupyter_start><jupyter_text>Confusion matrix F1 Score The F 1 score favors classifiers that have similar precision and recall. This is not alwayswhat you want: in some contexts you mostly care about precision, and in other con‐texts you really care about recall. For example, if you trained a classifier to detect vid‐eos that are safe for kids, you would probably prefer a classifier that rejects manygood videos (low recall) but keeps only safe ones (high precision), rather than a clas‐sifier that has a much higher recall but lets a few really bad videos show up in yourproduct (in such cases, you may even want to add a human pipeline to check the clas‐sifier’s video selection). On the other hand, suppose you train a classifier to detectshoplifters on surveillance images: it is probably fine if your classifier has only 30%precision as long as it has 99% recall (sure, the security guards will get a few falsealerts, but almost all shoplifters will get caught).Unfortunately, you can’t have it both ways: increasing precision reduces recall, andvice versa. This is called the precision/recall tradeoff.  Precision/Recall Trade off o understand this tradeoff, let’s look at how the SGDClassifier makes its classifica‐tion decisions. For each instance, it computes a score based on a decision function,and if that score is greater than a threshold, it assigns the instance to the positiveclass, or else it assigns it to the negative class. Figure 3-3 shows a few digits positionedfrom the lowest score on the left to the highest score on the right. Suppose the deci‐sion threshold is positioned at the central arrow (between the two 5s): you will find 4true positives (actual 5s) on the right of that threshold, and one false positive (actuallya 6). Therefore, with that threshold, the precision is 80% (4 out of 5). But out of 6actual 5s, the classifier only detects 4, so the recall is 67% (4 out of 6). Now if youraise the threshold (move it to the arrow on the right), the false positive (the 6)becomes a true negative, thereby increasing precision (up to 100% in this case), butone true positive becomes a false negative, decreasing recall down to 50%. Conversely,lowering the threshold increases recall and reduces precision.<jupyter_code>y_scores = sgd_clf.decision_function([some_digit])
y_scores
threshold = 0
y_some_digit_pred = (y_scores > threshold)
y_some_digit_pred
threshold = 10000
y_some_digit_pred = (y_scores > threshold)
y_some_digit_pred<jupyter_output><empty_output><jupyter_text>This confirms that raising the threshold decreases recall. The image actually repre‐sents a 5, and the classifier detects it when the threshold is 0, but it misses it when thethreshold is increased to 8,000.Now how do you decide which threshold to use? For this you will first need to get thescores of all instances in the training set using the cross_val_predict() functionagain, but this time specifying that you want it to return decision scores instead ofpredictions<jupyter_code>y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, method="decision_function")
from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)
def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):
    plt.plot(thresholds, precisions[:-1], "b--", label="Precision", linewidth=2)
    plt.plot(thresholds, recalls[:-1], "g-", label="Recall", linewidth=2)
    plt.legend(loc="center right", fontsize=16) # Not shown in the book
    plt.xlabel("Threshold", fontsize=16)        # Not shown
    plt.grid(True)                              # Not shown
    plt.axis([-50000, 50000, 0, 1])             # Not shown



recall_90_precision = recalls[np.argmax(precisions >= 0.90)]
threshold_90_precision = thresholds[np.argmax(precisions >= 0.90)]


plt.figure(figsize=(8, 4))                                                                
plot_precision_recall_vs_threshold(precisions, recalls, thresholds)
plt.plot([threshold_90_precision, threshold_90_precision], [0., 0.9], "r:")                 
plt.plot([-50000, threshold_90_precision], [0.9, 0.9], "r:")                               
plt.plot([-50000, threshold_90_precision], [recall_90_precision, recall_90_precision], "r:")
plt.plot([threshold_90_precision], [0.9], "ro")                                             
plt.plot([threshold_90_precision], [recall_90_precision], "ro")                                                                           # Not shown
plt.show()


threshold_90_precision = thresholds[np.argmax(precision>=0.90)]
y_train_pred_90 = (y_scores >= threshold_90_precision)
precision_score(y_train_5, y_train_pred_90)
recall_score(y_train_5, y_train_pred_90)<jupyter_output><empty_output><jupyter_text>The ROC curve  The receiver operating characteristic (ROC) curve is another common tool used withbinary classifiers. It is very similar to the precision/recall curve, but instead of plot‐ting precision versus recall, the ROC curve plots the true positive rate (another namefor recall) against the false positive rate. The FPR is the ratio of negative instances thatare incorrectly classified as positive. It is equal to one minus the true negative rate,which is the ratio of negative instances that are correctly classified as negative. TheTNR is also called specificity. Hence the ROC curve plots sensitivity (recall) versus1 – specificity.To plot the ROC curve, you first need to compute the TPR and FPR for various thres‐hold values, using the roc_curve() function<jupyter_code>from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_train_5, y_scores)
def plot_roc_curve(fpr, tpr, label=None):
    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0, 1], [0, 1], 'k--') # dashed diagonal
    plt.axis([0, 1, 0, 1])                                    
    plt.xlabel('False Positive Rate (Fall-Out)', fontsize=16) 
    plt.ylabel('True Positive Rate (Recall)', fontsize=16)    
    plt.grid(True)                                            

plt.figure(figsize=(8, 6))                         
plot_roc_curve(fpr, tpr)
plt.plot([4.837e-3, 4.837e-3], [0., 0.4368], "r:") 
plt.plot([0.0, 4.837e-3], [0.4368, 0.4368], "r:")  
plt.plot([4.837e-3], [0.4368], "ro")                                        
plt.show()


from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_5, y_scores)<jupyter_output><empty_output><jupyter_text>Since the ROC curve is so similar to the precision/recall (or PR)curve, you may wonder how to decide which one to use. As a ruleof thumb, you should prefer the PR curve whenever the positiveclass is rare or when you care more about the false positives thanthe false negatives, and the ROC curve otherwise. For example,looking at the previous ROC curve (and the ROC AUC score), youmay think that the classifier is really good. But this is mostlybecause there are few positives (5s) compared to the negatives(non-5s). In contrast, the PR curve makes it clear that the classifierhas room for improvement (the curve could be closer to the top-right corner).<jupyter_code>from sklearn.ensemble import RandomForestClassifier
forest_clf = RandomForestClassifier(random_state=42)
y_probas_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3, method="predict_proba")
y_scores_forest = y_probas_forest[:,1] #score = Proba of positive class
fpr_forest, tpr_forest, threshold_forest = roc_curve(y_train_5, y_scores_forest)
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, "b:", linewidth=2, label="SGD")
plot_roc_curve(fpr_forest, tpr_forest, "Random Forest")
plt.plot([4.837e-3, 4.837e-3], [0., 0.4368], "r:")
plt.plot([0.0, 4.837e-3], [0.4368, 0.4368], "r:")
plt.plot([4.837e-3], [0.4368], "ro")
plt.plot([4.837e-3, 4.837e-3], [0., 0.9487], "r:")
plt.plot([4.837e-3], [0.9487], "ro")
plt.grid(True)
plt.legend(loc="lower right", fontsize=16)
plt.show()
roc_auc_score(y_train_5, y_scores_forest)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08853118712273642, 'fraction_numerical': 0.03420523138832998, 'mean_word_length': 4.137596899224806, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '110226', 'license': 'MIT', 'n_tokens_mistral': 2605, 'n_tokens_neox': 2301, 'n_words': 896, 'path': 'classification.ipynb', 'repo_name': 'RajputJay41/Machine-Learning-from-scratch', 'size': '7952'}