<jupyter_start><jupyter_text>The first classifier we'll check is a simple linear classifier. These run quickly and sometimes work well.  As we know that the labels should always be integers, we will round the label outputs. We will not be performing any kind of dimension reduction to get at the more important components of the data and will expect a pretty poor fit here, as the features will tend to covary, and the model may tend to overfit.<jupyter_code>from sklearn import linear_model

LR_model = linear_model.LinearRegression()
LR_model.fit(train_X_sc,train_y)
print("The GOF score of this fit is: ", LR_model.score(train_X_sc,train_y))
label_pred = np.round(LR_model.predict(train_X_sc))

sns.boxplot(train_y,label_pred)
plt.title("Predicted V. Actual Labels of the Linear Classifier on the Training Set")
plt.ylabel("Predicted Label")
plt.xlabel("Actual Label")
plt.yticks(np.arange(1,10,1))
plt.show()
print("The accuracy of this fit is: ", np.mean(train_y==label_pred))<jupyter_output>The GOF score of this fit is:  0.61603856192773<jupyter_text>We can clearly see that the spread about each label is very large and while some numbers (see 1 and 4) have medians at their target values, there are a large number of outliers. With such poor accuracy on a training set, we expect this model to perform equally poorly on a test set. Although the fit to the training data is a poor one and we have little hope of capturing the test data in any meaningful way, we will quantify this result:<jupyter_code>print("The GOF score of this fit is: ", LR_model.score(train_X_sc,train_y))
label_pred = np.round(LR_model.predict(test_X_sc))

sns.boxplot(test_y,label_pred)
plt.title("Predicted V. Actual Labels of the Linear Classifier on the Test Set")
plt.ylabel("Predicted Label")
plt.xlabel("Actual Label")
plt.yticks(np.arange(1,10,1))
plt.show()
print("The accuracy of this fit is: ", np.mean(test_y==label_pred))<jupyter_output>The GOF score of this fit is:  0.61603856192773<jupyter_text>This exceptionally poor fitting of the test data suggests some overfitting of unimportant features for the training data. Therefore, we will move on to a more complex linear classifier developed to address issues like these: a ridge regression model. The ridge classifier we choose includes cross-validation on a number of penalties that may be imposed on coefficients. This should allow a better fit than if we were to simple guess at penalties.<jupyter_code># let's try a more complex linear model
ridge = linear_model.RidgeCV(alphas = np.logspace(-10,0,20))
ridge.fit(train_X_sc,train_y)
print("The GOF score for this regression is: ", ridge.score(train_X_sc,train_y))
label_pred_ridge = np.round(ridge.predict(train_X_sc))

sns.boxplot(train_y,label_pred_ridge)
plt.title("Predicted V. Actual Labels of the Ridge Linear Classifier on the Training Set")
plt.ylabel("Predicted Label")
plt.xlabel("Actual Label")

print("The accuracy of this fit is: ", np.mean(label_pred_ridge==train_y))<jupyter_output>The GOF score for this regression is:  0.6161201187462169
The accuracy of this fit is:  0.2265<jupyter_text>Again we see extremely poor performance on the training set. However, the classifier's penalties on coefficient size should allow a better fit to tet data.<jupyter_code>ridge = linear_model.RidgeCV(alphas = np.logspace(-10,0,20))
ridge.fit(train_X_sc,train_y)
print("The GOF score for this regression is: ", ridge.score(test_X_sc,test_y))
label_pred_ridge = np.round(ridge.predict(test_X_sc))

sns.boxplot(test_y,label_pred_ridge)
plt.title("Predicted V. Actual Labels of the Ridge Linear Classifier on the Test Set")
plt.ylabel("Predicted Label")
plt.xlabel("Actual Label")

print("The accuracy of this fit is: ", np.mean(label_pred_ridge==test_y))<jupyter_output>The GOF score for this regression is:  0.6230985236031041
The accuracy of this fit is:  0.2212<jupyter_text>As anticipated, we see a fit of the test set that much more closely resembles the (poor) fit of the training data. Although a poor fit, this example does illustrate some of the ridge classifier's strength over a simple linear classifier.As noted above, one of the difficulties of performing linear analysis on this set, up to this point, has been a lack of dimension reduction. We will, therefore, be making one final attempt at linear classification using dimension reduction. To do so, we will employ Linear Discrimination Analysis. The LDA method performs dimension reduction by any of 3 different means and using some number of dimensions that are either set by the user or set by default. In order to maximize our results, we will perform Singular Value Decomposition on the training set to find those components that will provide the most relevant results. We will be using an approximation to the truncated SVD decomposition method that significantly improves on computation time while sacrificing minimal accuracy.<jupyter_code>from sklearn.utils.extmath import randomized_svd
u, s, vt = randomized_svd(train_X_sc, n_components = 9)
print("There are ", np.shape(s[s>=np.max(s)*0.75]), "components with singular values greater than or equal to 75% of the maximum singular value")
# By default, the number of components the classifier would use is 9 (number of classes - 1).
# We will use 3 instead.

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
LDA = LinearDiscriminantAnalysis(n_components = 3)
LDA.fit(train_X_sc,train_y)
print("The GOF score for this fit is: ", LDA.score(train_X_sc,train_y))
label_pred_LDA = LDA.predict(train_X_sc)

sns.boxplot(train_y,label_pred_LDA)
plt.title("Predicted V. Actual Labels of the LDA Linear Classifier on the Training Set")
plt.ylabel("Predicted Label")
plt.xlabel("Actual Label")
plt.yticks(np.arange(0,10,1))

print("The accuracy of this fit is: ", np.mean(label_pred_LDA==train_y))<jupyter_output>The GOF score for this fit is:  0.8714833333333334
The accuracy of this fit is:  0.8714833333333334<jupyter_text>This is already a significant improvement on our previous classifiers, with only outliers being misclassified. However, it's clear there are still a large number of outliers. Now we will check it against the test set.<jupyter_code>print("The GOF score for this fit is: ", LDA.score(test_X_sc,test_y))
label_pred_LDA = LDA.predict(test_X_sc)

sns.boxplot(test_y,label_pred_LDA)
plt.title("Predicted V. Actual Labels of the LDA Linear Classifier on the Test Set")
plt.ylabel("Predicted Label")
plt.xlabel("Actual Label")
plt.yticks(np.arange(0,10,1))

print("The accuracy of this fit is: ", np.mean(label_pred_LDA==test_y))<jupyter_output>The GOF score for this fit is:  0.873
The accuracy of this fit is:  0.873<jupyter_text>Somewhat to my surprise, this method offered a better fit of the test set than the training set, with fewer outliers and a 0.15% accuracy improvement. However, suppose the nuclear launch codes depend on our ability to classify MNIST. To the end of minimizing our chances of erroneously destroying the planet, let us try to improve on these methods.From here, we can move on to a simpler non-linear classifier: the Nearest Neighbor classifier. The simple concept is to match each test set's predicted label to the label of the training example from which it is separated in Euclidean space by a minimal vector.We will start with mapping to the single nearest neighbor. Although it is possible to perform this operation using NearestNeighbors, we will instead use the KNeighborsClassifier for its built in R^2 scoring method, which provides a more direct comparison between it and previous methods.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

kneigh = KNeighborsClassifier(n_neighbors=1)
kneigh.fit(train_X_sc, train_y)
print("The GOF score for this regression is: ",kneigh.score(test_X_sc,test_y))

y_lab = kneigh.predict(test_X_sc)
y_actual = test_y
print("The Accuracy of this Classifier is: ", np.mean(y_lab==y_actual))
sns.boxplot(y_actual,y_lab)
plt.title("Predicted V. Actual Labels for Nearest Neighbor Classification")
plt.ylabel("Predicted Labels")
plt.xlabel("Actual Labels")
plt.show()<jupyter_output>The GOF score for this regression is:  0.9427
The Accuracy of this Classifier is:  0.9427<jupyter_text>We will make one further modification to our classifier for this notebook. Rather than relying on a single nearest neighbor or a small set of nearest neighbors, we will represent each class by its centroid, equivalent to the center of mass of a distributed body. From there, we will match each test example to the centroid to which it is closest in euclidean space. This should make the classifier somewhat less sensitive to outlier repreesntations in the training set. This may reduce error where examples in the training set are generally tightly clustered. However, if there are significant outliers in the training set, as we might expect based on the results of out KNeighbors approach, it might introduce errors by misclassifying outliers in the test set.<jupyter_code>from sklearn.neighbors import NearestCentroid

NC = NearestCentroid()
NC.fit(train_X_sc, train_y)
NC_lab_pred = NC.predict(test_X_sc)
print("The GOF score for this classifier is: ", NC.score(test_X_sc,test_y))

y_actual = test_y
print("The accuracy of this Classifier is: ", np.mean(NC_lab_pred==y_actual))
sns.boxplot(y_actual,NC_lab_pred)
plt.title("Predicted V. Actual Labels for Nearest Centroid Classification")
plt.ylabel("Predicted Labels")
plt.xlabel("Actual Labels")
plt.show()<jupyter_output>The GOF score for this classifier is:  0.8089
The accuracy of this Classifier is:  0.8089
{'fraction_non_alphanumeric': 0.07145837679949928, 'fraction_numerical': 0.018777383684539954, 'mean_word_length': 5.770480225988701, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '329400', 'license': 'CC0-1.0', 'n_tokens_mistral': 3048, 'n_tokens_neox': 2813, 'n_words': 1188, 'path': 'Classifying_MNIST_with_Basic_Methods.ipynb', 'repo_name': 'LenaRosendahl/Stats_and_ML_demo_repo', 'size': '9586'}