<jupyter_start><jupyter_text>Chapter 3: ClassificationIn this chapter the code will deal with classification techniques in ML.  MNISTWe will start off by using the MNIST dataset which is a combination of handwritten digits written by high school students and US Postal Service. This problem tries to recognize the handwritten digits.  Get the data We will use sklearn's `fetch_mldata` function to get the MNIST dataset. All of sklearn's datasets have a dictionary containing the following things.- Descr -- Description of the data- data -- The actual data- target -- The target variable<jupyter_code>from sklearn.datasets import fetch_mldata
mnist = fetch_mldata("MNIST Original")
mnist
X, y = mnist["data"], mnist["target"]
X.shape, y.shape<jupyter_output><empty_output><jupyter_text>Plotting the digitsIn this section we will try and plot a few of the digits to see what they actually look. The data is greyscale so it should be a black-white picture<jupyter_code>%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt

some_digit = X[36000]
some_digit_image = some_digit.reshape(28, 28)

plt.imshow(some_digit_image, cmap = matplotlib.cm.binary,
           interpolation="nearest")
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>Train Test splitThe MNIST dataset is already split into training and testing. The first 60k are training data and the last 10k are testing data. We will get the train and test splits and also shuffle the training data so that our CV folds find all of the digits in each of the fold.<jupyter_code>X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]
import numpy as np

shuffle_index = np.random.permutation(60000)
X_train, y_train = X_train[shuffle_index], y_train[shuffle_index]<jupyter_output><empty_output><jupyter_text>Binary PredictorBelow we will try to make the problem simple by just trying to predict whether the digit is a 5 or not. This is a binary classification problem where the classifier only tries to predict if the digit is a 5 or not.We will then calculate the accuracy of the model and identify the issue related to accuracy as a metric of goodness of fit in classification problems<jupyter_code>y_train_5 = (y_train == 5)  # True for all 5s, False for all other digits.
y_test_5 = (y_test == 5)
from sklearn.linear_model import SGDClassifier

sgd_clf = SGDClassifier(random_state=42)
sgd_clf.fit(X_train, y_train_5)
sgd_clf.predict([some_digit])
test_prediction = sgd_clf.predict(X_test)
sum(test_prediction == y_test_5)/float(len(y_test_5))<jupyter_output><empty_output><jupyter_text>Custom CrossvalidationSometimes we will want to find out a way to implement a custom cross validation function. We can do this easily using sklearn's inbuilt functions<jupyter_code>from sklearn.base import clone
from sklearn.cross_validation import StratifiedKFold

skfolds = StratifiedKFold(y_train_5, n_folds = 3, random_state=42)
fold = 0

for train_index, test_index in skfolds:
    fold = fold + 1
    clone_clf = clone(sgd_clf)
    X_train_fold = X_train[train_index]
    y_train_fold = y_train_5[train_index]
    X_test_fold = X_train[test_index]
    y_test_fold = y_train_5[test_index]
    
    clone_clf.fit(X_train_fold, y_train_fold)
    predict = clone_clf.predict(X_test_fold)
    
    n_correct = sum(predict == y_test_fold)
    print "Accuracy for fold %s = %s" %(fold, n_correct/(1.0*len(y_test_fold)))<jupyter_output>/Users/soutikchakraborty/venv/lib/python2.7/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)<jupyter_text>Cross ValidationHere we will use sklearns cross_val_score function to calculate the cross validated accuracy score of each fold<jupyter_code>from sklearn.cross_validation import cross_val_score

sgd_clf = SGDClassifier(random_state=42)
cross_val_score(sgd_clf, X_train, y_train_5, cv = 5, scoring = "accuracy")<jupyter_output><empty_output><jupyter_text>As we can see that the above prediction is at 96%. So it that a good predictor? Let us check the accuracy of a simple predictor that predicts that all of the values are not 5.<jupyter_code>not_5_predictor = False
n_correct = sum(not_5_predictor == y_test_5)
print "Accuracy is %s"%(n_correct/(1.0*len(y_test_5)))<jupyter_output>Accuracy is 0.9108<jupyter_text>As we can see that a simple classifier can acheive an accuracy of 91%. This makes sense as 10% of the data is 5 and the rest is not 5. This is a reason why accuracy is not used as a metric of goodness of fit.<jupyter_code>from sklearn.model_selection import cross_val_predict

y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)
from sklearn.metrics import confusion_matrix

confusion_matrix(y_train_5, y_train_pred)
from sklearn.metrics import precision_score, recall_score, f1_score

print "Precision of the model is : %s"%(precision_score(y_train_5, y_train_pred))
print "Recall of the model is : %s"%(recall_score(y_train_5, y_train_pred))
print "F1 of the model is : %s"%(f1_score(y_train_5, y_train_pred))<jupyter_output>Precision of the model is : 0.874369655777
Recall of the model is : 0.735657627744
F1 of the model is : 0.799038268884<jupyter_text>Recall score is the harmonic mean of the precision score and the recall score. It favors a model with similar precision and recall but this is not ideal as sometimes you would want a model to have high precision (Filter out adult videos for children) while sometimes a high recall score is needed (identifying burglars from video)  Precision/Recall tradeoffYou can't have a model that has high precision and high recall as increasing precision will cause lower recall and vice-versa this is known as the precision/recall tradeoff.We will now try and plot the precision and recall curves of our model<jupyter_code>y_score = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, method="decision_function")
from sklearn.metrics import precision_recall_curve

precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_score)
def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):
    plt.plot(thresholds, precisions[:-1], "b--", label="Precision")
    plt.plot(thresholds, recalls[:-1], "g-", label="Recall")
    plt.xlabel("Threshold")
    plt.legend(loc="center left")
    plt.ylim([0, 1])

plt.figure(figsize=[15, 5])
plot_precision_recall_vs_threshold(precisions, recalls, thresholds)
plt.show()
plt.figure(figsize=[15, 5])
plt.plot(recalls, precisions)
plt.ylabel("precisions")
plt.xlabel("recalls")
plt.show()<jupyter_output><empty_output><jupyter_text>ROC CurveWe will now plot the True Positive Rate v/s False Positive Rate against each other and create something known as ROC curve (Reciever Operator Characteristic). This curve is very similar to the precision/recall curve but it plot *sensitivity* (recall) vs *1-specificity*<jupyter_code>from sklearn.metrics import roc_curve
fpr, tpr, threshold = roc_curve(y_train_5, y_score)
def plot_roc_curve(fpr, tpr, label = None):
    plt.plot(fpr, tpr, label = label)
    plt.plot([0, 1], [0, 1], "k--")
    plt.axis([0, 1, 0, 1])
    plt.xlabel("False Positive Rate")
    plt.ylabel("True Positive Rate")
    
plt.figure(figsize=[15, 7])
plot_roc_curve(fpr, tpr)
plt.show()<jupyter_output><empty_output><jupyter_text>As we can see from above that the higher TPR (precision) the lower FPR (recall) vice-versa. The dotted line is the line that represents random prediction. It shows how the curve will look if a completely random process was used to predict the values.The other way to quantify ROC curves is to find the Area Under the Curve or AUC value. This is shown below.<jupyter_code>from sklearn.metrics import roc_auc_score

roc_auc_score(y_train_5, y_score)<jupyter_output><empty_output><jupyter_text>Random Forest ClassifierNow let us fit a random forest classifier and check its AUC and plot its ROC curve with the one obtained from Stocastic Gradient Descent model<jupyter_code>from sklearn.ensemble import RandomForestClassifier

forest_clf = RandomForestClassifier(random_state = 42)
y_proba_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3, method = "predict_proba")
y_score_forest = y_proba_forest[:, 1]

fpr_forest, tpr_forest, threshold_forest = roc_curve(y_train_5, y_score_forest)
# Plotting
plt.figure(figsize=[15, 7])
plt.plot(fpr, tpr, "b:", label = "SGD")
plot_roc_curve(fpr_forest, tpr_forest, label = "Random Forest")
plt.legend(loc = "lower right")
plt.show()
print "AUC value of random Forest : %s" %(roc_auc_score(y_train_5, y_score_forest))<jupyter_output>AUC value of random Forest : 0.992297580867<jupyter_text>As we can see that the Random Forest classifier is better than that of the SGD based classifier. We can dig further into the precision and recall curves to understand which one performs better in which condition.  Multi-class ClassificationLet us now try and predict all the values of the digits instead of just 5 or not 5. We will use SGD and RandomForest models and try and fit/predict on our train dataset and evaluate the performance of the models.<jupyter_code># SGD
sgd_clf.fit(X_train, y_train)
sgd_clf.predict([some_digit])
# Random Forest
forest_clf.fit(X_train, y_train)
forest_clf.predict([some_digit])
forest_clf.predict_proba([some_digit])
# Cross validated score of RandomForest
cross_val_score(sgd_clf, X_train, y_train, cv = 3, scoring = "accuracy")<jupyter_output><empty_output><jupyter_text>As we can see that the model only reaches 84% accuracy. We can improve this accuracy by standardizing the data.<jupyter_code>from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))

cross_val_score(sgd_clf, X_train_scaled, y_train, cv = 3, scoring = "accuracy")<jupyter_output><empty_output><jupyter_text>We could easily reach 90% accuracy by just standardizing our data<jupyter_code>cross_val_score(forest_clf, X_train_scaled, y_train, cv = 3, scoring = "accuracy")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07807196198234895, 'fraction_numerical': 0.02017263117059451, 'mean_word_length': 5.944107744107744, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '445549', 'license': 'MIT', 'n_tokens_mistral': 3542, 'n_tokens_neox': 3288, 'n_words': 1137, 'path': '3. Classification.ipynb', 'repo_name': 'soutik/Hands-On-Machine-Learning-with-ScikitLearn-and-TensorFlow', 'size': '10311'}