<jupyter_start><jupyter_text>Multiclass ClassificationWhile binary classifiers are used to distinguish between two classes (e.g. detect if a transaction is a fraudulent one, classify an email into either spam or non-spam and etc.), multiclass classifiers distinguish between more than two classes. For example, we want to predict a picture of the <NAME> into a set of predefined species. While some algorithms (e.g. K nearest neighbors and naive bayes classifier) can extend to handle multiple classes naturally, others are less obvious (e.g. Support Vector Machine). There are various ways that we can use to perform multiclass classification by leveraging any binary classifiers. In this exercise, you will implement two such strategies for multiclass classification: _One-versus-all_ strategy and _One-versus-one_ strategy.- **One-versus-all (OvA)**: In this strategy, we train a single binary classifier per class, with the samples of that class as positive samples and all other samples as negatives. During inference, we get the prediction from each classifier and select the class with the highest score. This strategy is also called the one-versus-the-rest strtegey. - **One-versus-one (OvO)**: In this strategy, we train a binary classifier for every pair of classes. If there are N classes in the problem, you need to train N * (N-1) / 2 classifiers. During inference, we have to run through all N * (N-1) / 2 classifiers and ses which class wins the most votes. The main advantage of OvO strategy is that each binary classifier only needs to be train on the part of the training dataset for the two classes that it needs to separate.<jupyter_code># import packages
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn import datasets
from sklearn.linear_model import LogisticRegression

# make this notebook's output stable across runs
np.random.seed(0)<jupyter_output><empty_output><jupyter_text>MNIST DatasetIn this lab assignment, we use the classic machine learning benchmark dataset **MNIST** assembed by the National Institute of Standards and Technology in the 1980s. The MNIST dataset includes 60K training and 10K test images of grayscale handwriting images and the task is to classify these images into 10 classes (0 through 9).Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.Both the train and test CSV files have 785 columns. The first column, called "label", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.Each pixel column in the training set has a name like pixelx, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. Then pixelx is located on row i and column j of a 28 x 28 matrix, (indexing by zero).<jupyter_code># Load train and test data from CSV files.
train = pd.read_csv("mnist_train.csv", header=None)
test = pd.read_csv("mnist_test.csv", header=None)

train_x = train.iloc[:,1:]
train_y = train.iloc[:,0]

test_x = test.iloc[:,1:]
test_y = test.iloc[:,0]
# Output the number of images in each class in the train and test datasets.
print(train_y.value_counts().sort_index())
print(test_y.value_counts().sort_index())<jupyter_output><empty_output><jupyter_text>Here we define two helper functions which plot the image of an instance for data exploration and debugging.<jupyter_code>def plot_digit(x, y):
    """
    Plot a single image of the MNIST dataset
    """
    plt.imshow(np.reshape(x.values, (28, 28)), cmap='gray_r')
    plt.title('Digit Label: {}'.format(y))
    plt.show()

def plot_digits(x, y):
    """
    Plot the image of the digits in MNIST dataset
    """
    plt.clf()
    plt.style.use('seaborn-muted')

    fig, axes = plt.subplots(
    3,3, 
    figsize=(5,5),
    sharex=True, sharey=True,
    subplot_kw=dict(adjustable='box-forced', aspect='equal'))

    for i in range(0, 9):
        # Idnetify the subgraph given class i.
        subplot_row = i // 3 
        subplot_col = i % 3 
        ax = axes[subplot_row, subplot_col]

        # Plot image on subplot.
        ax.imshow(np.reshape(x.iloc[i,:].values, (28, 28)), cmap='gray_r')

        ax.set_title('Digit Label: {}'.format(y[i]))
        ax.set_xbound([0,28])

    plt.tight_layout()
    plt.show()<jupyter_output><empty_output><jupyter_text>Plot a single image in MNIST dataset<jupyter_code>idx = 3
plot_digit(train_x.iloc[idx,:], train_y[idx])<jupyter_output><empty_output><jupyter_text>View the digit images from class 1 to 9.To better understand the MNIST dataset, we plot the first occurence of each class (1-9) in the train dataset using the grayscale inputs. In the following code, we plot nine images of digits 0 in the train dataset.<jupyter_code># Find the first 9 occurrences of digit zero in the train dataset
zero_digit_idx = list(train_y[train_y == 0].index[0:9])

# Plot digits
plot_digits(train_x.iloc[zero_digit_idx,:], train_y[zero_digit_idx].values)<jupyter_output><empty_output><jupyter_text>**Coding Part 1: Plot the first image of digits 1 to 9 in the training dataset**<jupyter_code># Find the first occurrence of each class in the train dataset
idx = [-1] * 9
for i in range(1, 10):
  # Find the first image from the class i.
  idx[i-1] = train_y[train_y == i].index[0]

# Plot digits
plot_digits(train_x.iloc[idx,:], train_y[idx].values)<jupyter_output><empty_output><jupyter_text>---  Classify even digits.To reduce the computational assumption, we simplify the problem to classify only the odd numbers, ie. 1, 3, 5, 7, and 9, instead of all 10 digits.<jupyter_code># helper functions
def sample_data(x, y, digits):
    """
    Sample a subset of all the images based on the digits provided
    """
    idx = []
    for i in digits:
        idx = idx + list(y.index[y == i])

    return (x.iloc[idx,:], y.iloc[idx])

def evaluate(y, z):
    """
    Evaluate the classification accuracy and also generate the confusion matrix. 
    """
    confusion_matrix = pd.crosstab(y, z, rownames=['Actual'], colnames=['Predicted'])
    accuracy = sum(np.diag(confusion_matrix)) / confusion_matrix.values.sum()

    return (accuracy, confusion_matrix)
# generate the final dataset for model training for the given digits
digits = [1, 3, 5, 7, 9]
tr_x, tr_y = sample_data(train_x, train_y, digits)
te_x, te_y = sample_data(test_x, test_y, digits)<jupyter_output><empty_output><jupyter_text>One-versus-all (OvA) strategy**Coding Part 2: In this session, you will implement the OvA stragety for multiclass classification using logistic regression as the base binary classifier. After the implementation, you will apply it to the MNIST dataset and see its performance.**<jupyter_code>class OvA():
    """
    OvA strategy for multiclass classification.
    """

    def __int__(self):

    def train(self, x, y):
        """
        TODO: Train the multiclass classifier using OvA strategy. 
        """

    def predict(self, x):
        """
        TODO: Make predictions on multiclass problems using the OvA strategy. 
        """<jupyter_output><empty_output><jupyter_text>Train the OvA multiclass classifier on MNIST dataset<jupyter_code>ova = OvA()
ova.train(tr_x, tr_y)<jupyter_output><empty_output><jupyter_text>Predict on the test dataset using OvA multiclass classifier<jupyter_code>te_z_ova = ova.predict(te_x)<jupyter_output><empty_output><jupyter_text>Evaluate the performance of OvA<jupyter_code>ova_accuracy, ova_confuction_matrix = evaluate(te_y.values, te_z_ova)
print("Accuracy of OvA classifier is {}.".format(ova_accuracy))
print("Confusion matrix of OvA classifier: \n {}".format(ova_confuction_matrix))<jupyter_output><empty_output><jupyter_text>---  One-versus-one (OvO) strategy**Coding Part 3: In this session, you will implement the OvO stragety for multiclass classification using logistic regression as the base binary classifier. After the implementation, you will apply it to the MNIST dataset and see its performance.**<jupyter_code>class OvO():
    """
    OvO strategy for multiclass classification.
    """

    def __int__(self):

    def train(self, x, y):
        """
        TODO: Train the multiclass classifier using OvO strategy. 
        """

    def predict(self, x):
        """
        TODO: Make predictions on multiclass problems using the OvO strategy. 
        """<jupyter_output><empty_output><jupyter_text>Train the OvO multiclass classifier on MNIST dataset<jupyter_code>ovo = OvO()
ovo.train(tr_x, tr_y)<jupyter_output><empty_output><jupyter_text>Predict on the test dataset using OvO multiclass classifier<jupyter_code>te_z_ovo = ovo.predict(te_x)<jupyter_output><empty_output><jupyter_text>Evaluate the performance of OvO<jupyter_code>ovo_accuracy, ovo_confuction_matrix = evaluate(te_y.values, te_z_ovo)
print("Accuracy of OvO classifier is {}.".format(ovo_accuracy))
print("Confusion matrix of OvO classifier: \n {}".format(ovo_confuction_matrix))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09023285899094437, 'fraction_numerical': 0.010888313928417421, 'mean_word_length': 4.531902206320811, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '73270', 'license': 'MIT', 'n_tokens_mistral': 2995, 'n_tokens_neox': 2784, 'n_words': 1078, 'path': 'module01/resources/HW01.ipynb', 'repo_name': 'kegedy/MLEARN520', 'size': '9281'}