<jupyter_start><jupyter_text>> Note: Always open in Colab for the best learning experience.  Classification: Softmax RegressionIn the previos lesson, we learned about logistic regression and how it can be used to construct a binary classifier to distinguish between two categorical values (e.g., whether a photo is of a dog or a cat). Logistic regression is great when we are working with only two categorical values. In practice, we are often more interested in differentiating between many categorical values (e.g., more than 2 classes):- Is this image of a cat, a dog, or a chicken?- Is this song in the genre of hip hop, pop, funk, rock, etc.?- What brand of clothing does this t-shirt belong to?When we want to distinguish between many classes (called *multi-class classification*), we can use a classification technique called softmax regression.In this notebook, we will learn the foundations of softmax regression and demonstrate how to solve multi-class classification problems using an example--building a softmax regression classifier to distinguish 10 handwritten digit (0-9). The ideas we introduce here will build on previous material and continue to lay out the fundamental concepts used in deep learning and neural networks, which we will cover in future lessons. Here is the lesson roadmap:1. Review: representing categorical data2. Introduction to softmax regression3. Building a softmax regression classifier: recognize 10 handwritten digits from the MNIST dataset 4. Summary  Review: representing categorical data    Representing data: a Shina Inu, Retriever, and LabTo motivate our understanding of softmax regression, let's consider the example where we want to distinguish 3 different dog breeds--(golden) retrievers, labs, and shiba inus, given $2 \times 2$ grayscale images. We can represent each pixel value with a single scalar (number), giving us four features  $x_1,x_2,x_3,x_4$. Further, we know that each image belongs to one among the categories "retriever", "lab", "shina inu".To make the categorical labels useful, we need to convert it into a numerical representation.  There are two general ways to represent the categorical data in numeric terms. The first way is to choose choose $y \in \{1, 2, 3\}$, where the integers represent {retriever, lab, shina inu} repectively. But, as you learned previously, the second way is better: *one-hot encoding*. As a refresher, a one-hot encoding is a vector with as many components as we have categories. The component corresponding to particular sample's category is set to 1 and all other components are set to 0. So in our case, this translates to:$$y \in \{ (1, 0, 0), (0, 1, 0), (0, 0, 1) \},$$where $y$ would be a three-dimensional vector representing the dog breeds, with $(1, 0, 0)$ corresponding to "retriever", (0, 1, 0) to "lab", and (0, 0, 1) to "shiba inu".  Intro to softmax regression    Softmax regression is a single-layer neural network | Source: Dive Into Deep LearningNow that we have a healthy understanding of classification techniques like *one-hot encoding* and logistic regression, let's dive into the softmax regression.Softmax regression is perhaps the most common machine learning algorithm. It is a special case of logistic regression where the labels ($y$) is categorical in nature but there are many categories rather than simply two. It is called "softmax" regression because it uses a *logit* function, called the *softmax* function, to estimate the *conditional probability* of a given class among many classes (i.e., more than 2). Unlike linear and logistic regression, softmax regression requires a model with multiple outputs, one per class. To address multi-class classification with softmax regression classifiers, we will need as many linear functions as we have outputs. Each output will correspond to its own linear function. In our case, since we have 4 features and 3 possible output categories, we will need 12 scalars to represent the weights, ( $w$  with subscripts) and 3 scalars to represent the biases ($b$ with subscripts). We compute these three *logits*, $o_1,o_2$, and $o_3$, for each input:\begin{split}\begin{aligned}o_1 &= x_1 w_{11} + x_2 w_{12} + x_3 w_{13} + x_4 w_{14} + b_1,\\o_2 &= x_1 w_{21} + x_2 w_{22} + x_3 w_{23} + x_4 w_{24} + b_2,\\o_3 &= x_1 w_{31} + x_2 w_{32} + x_3 w_{33} + x_4 w_{34} + b_3.\end{aligned}\end{split}We can depict this calculation with the neural network diagram shown above. Just as in linear regression, softmax regression is also a *single-layer neural network*. And since the calculation of each output, $o_1,o_2$, and $o_3$, depends on all inputs, $x_1, x_2, x_3$, and $x_4$, the output layer of softmax regression can also be described as *fully-connected* layer.To express the model more compactly, we can use linear algebra notation. In vector form, we arrive at  $\mathbf{o} = \mathbf{W} \mathbf{x} + \mathbf{b} $, a form better suited both for mathematics, and for writing code. Note that we have gathered all of our weights into a  $3 \times 4$  matrix and that for a given example $\mathbf{x}$, our outputs are given by a *matrix-vector product* of our weights by our inputs plus our biases  $b$. From the above describtion, hopefully softmax regression feels suprisingly familiar to you, since it shares many of the techniques used by linear and logistic regression methods.  The softmax operation    Single-layer softmax regression neural network | Source: deepnotes.ioThe main approach of softmax regression is to interpret the outputs of our model as probabilities. We will *optimize* our parameters to produce probabilities that *maximize the likelihood* of the observed data, just like logistic regression. Then, we generate predictions using a threshold (which we define), for example, choosing the *argmax* of the predicted probabilities. The *argmax* defines a given sample's predicted class based on the category with the highest probability value.Put formally, we would like outputs  $\hat{y}_k$  that we can interpret as the probability that a given item belongs to class $k$. Then, we can choose the class with the largest output value as our prediction  $\text{argmax}_k y_k $. For example, if $\hat{y}_1 , \hat{y}_2$, and $\hat{y}_3$  are  $0.1, 0.8$, and $0.1$, respectively, then we predict category $2$, which (in our example) represents "lab".To interpret our outputs as probabilities, we must guarantee two things:1. They will be nonnegative.2. Given an input sample, the total of all the class probabilities sum up to 1.To transform our logits such that they become nonnegative and sum to $1$, while maintaining that the model remains differentiable (for gradient descent), we first exponentiate each logit (ensuring non-negativity) and then divide by their sum (ensuring that they sum to $1$). $$\hat{\mathbf{y}} = \mathrm{softmax}(\mathbf{o})\quad \text{where}\quad\hat{y}_i = \frac{\exp(o_i)}{\sum_j \exp(o_j)}.$$Now we can guarantee $\hat{y}_1+\hat{y}_2+\hat{y}_3 = 1$  with  $0≤\hat{y}_i≤1$ for all $i$ . Thus, we can interpret $\hat{y}$ as a proper probability distribution. Note that the softmax operation does not change the ordering among the logits, and thus we can still pick out the most likely class by:$$\hat{\imath}(\mathbf{o}) = \operatorname*{argmax}_i o_i = \operatorname*{argmax}_i \hat y_i.$$  A note on the loss functionThe loss function for softmax regression is called categorical *cross-entropy*. A detailed description of cross-entropy is beyond the scope of this lesson. However, it's worth a quick summary. At a high level, cross-entropy is a measure of the difference between two probability distributions. It measures the amount of information (also called *bits*) needed to encode the data given our model. The goal is to predict the correct labels most of the time (via *maximum likelihood estimation*), while minimizing the *suprise* (entropy) required to communicate the labels.  Summary: Softmax RegressionTo summarize softmax regression:- Category labels ($y$) are converted to discrete integer values and represented using multi-dimensional vectors (e.g., dimensions $=$  of classes $k$).- The *softmax* logit function maps input features ($\mathbf{x}$) to probabilities and guarantees that they are nonnegative and sum up to 1.- A category prediction is determined by the threshold function, such as *argmax*, which selects the class with the highest probability among all $k$ classes.   - Softmax regression classifiers try to *maximize the likelihood* of the observed data, like logistic regression.- Categorical *cross-entorpy* is the loss function for softmax regression.    Softmax Regression: build a classifier to recognize handwritten digitsNow that we know about the fundamentals of softmax regression, let's apply this method to a real-world problem--distinguishing between 10 handwritten digits (0-9) given the MNIST dataset--a dataset of grayscale handwritten digits. In this section, we will demonstrate in an end-to-end fashion the process of creating a softmax regression classifier: from building, to training, and finally evaluating the model to solve the handwritten digit classification task. This process involves several steps:1. Find, load, and prepare a dataset for the model.2. Build the model.3. Train the model using an algorithm such as stochastic gradient descent.4. Evaluate the quality of our model.5. Draw conclusions.<jupyter_code># import torch
import torch
import torch.nn as nn

# import torchvision datasets
import torchvision.datasets

# Commonly used modules
import numpy as np
import os
import sys

# Images, plots, display, and visualization
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import cv2
import IPython
from six.moves import urllib<jupyter_output><empty_output><jupyter_text>1. Finding and preparing the datasetFor step 1, we found the [MNIST dataset](http://yann.lecun.com/exdb/mnist/). The dataset contains 70k standardized grayscale images of handwritten digits at a resolution of $28 \times 28$ pixels. Our goal is to build a classification model to take one of these images as input and predict the most likely digit contained in the image (along with a relative confidence about the prediction):Source: MIT Deep LearningLoading the dataset will return two PyTorch Datasets:* The `training_data` dataset contains both matching training images and their labels-—the data the model uses to learn.* The `test_data` dataset contains both matching test images and their labels--the data the model is tested on.The images are $1\times28\times28$ PyTorch tensors (i.e., the x variables), with pixel values ranging between 0 and 1(usually 0-255 but the pixels were already standardized). The *labels* (i.e., $y$) are an array of integers, ranging from 0 to 9. Usually we have to use *one-hot encoding* (the technique we learned about in the logistic regression lesson) to convert these labels to vectors (i.e., arrays with mostly 0s and a 1 at the index that corresponds to the data sample's digit category), however PyTorch does this for us with the loss function. The images are *standardized* already by PyTorch. This is achieved by dividing the pixels of each image by 255 (i.e., the max pixel value). Standardizing the data encourages our model to learn more generalizable features and helps it perform better on outside data. The final data processing step is "flattening" the $28\times28$ image pixel matrices into 784 image pixel arrays, and this is done when the image is passed in the model. We reshape the image matrices into arrays because our model expects the input to be a vector/array with 784 features (i.e., values). Now, let's load the data!<jupyter_code># Downloading the data
root = '/content/MNIST'
training_data = torchvision.datasets.MNIST(root, 
                                           train=True, 
                                           download=True,
                                           transform=torchvision.transforms.ToTensor())
test_data = torchvision.datasets.MNIST(root, 
                                           train=False, 
                                           download=True,
                                           transform=torchvision.transforms.ToTensor())

# Loading Data
training_dataloader = torch.utils.data.DataLoader(training_data,
                                                  batch_size=2048,
                                                  shuffle=True)

test_dataloader = torch.utils.data.DataLoader(test_data,
                                                  batch_size=2048,
                                                  shuffle=False)

print('Length of training dataset: {}'.format(len(training_data)))
print('Length of test dataset: {}'.format(len(test_data)))
images, labels = next(iter(training_dataloader))

print('\nTraining batch feature dimension: {}'.format(images.shape))
print('Testing batch feature dimension: {}'.format(labels.shape))
print('\nFirst 5 labels:', labels[:5])<jupyter_output>Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /content/MNIST/MNIST/raw/train-images-idx3-ubyte.gz<jupyter_text>Let's display the first 5 images from the *training set* and display the class name below each image.<jupyter_code>plt.figure(figsize=(10,2))
for i in range(5):
    plt.subplot(1,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(training_data[i][0].numpy().reshape(28, 28), cmap=plt.cm.binary)
    plt.xlabel(training_data[i][1])<jupyter_output><empty_output><jupyter_text>3. Build the modelNow that the data is ready, we can build the softmax classifier. We will use PyTorch to define a simple softmax regression model (single-layer fully-connected neural network) to predict the class of each image (a digit between 0 and 9). Given a sample with a corresponding set of class predictions, our model will select the class with the highest prediction value using *argmax*. We also define the loss function and optimization algorithm to compute gradients and give a score to the model. We will use *category cross-entropy* loss, *stochastic gradient descent*, and track the *accuracy* metric.Notice that the second parameter in `nn.Linear` layer contains 10 *neurons* instead of 1, which is different from the logistic regression lesson (1 neuron). This is because we need to calculate probabilities for 10 classes (digits 0-9).<jupyter_code>class Softmax_Model(nn.Module):
    # Contructor
    def __init__(self, num_classes):
        super(Softmax_Model, self).__init__()

        # Defining Fully-Connected Layer
        self.fc = nn.Linear(28*28, num_classes) # 28*28 since each image is 28*28

    def forward(self, x):
        
        # Need to flatten each image in the batch
        x = x.flatten(start_dim=1)

        # Input it into the FC layer
        x = self.fc(x)

        return x

num_classes = 10
model = Softmax_Model(num_classes)

# Defining loss function
criterion = torch.nn.CrossEntropyLoss()

# Defining optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
print(model)<jupyter_output>Softmax_Model(
  (fc): Linear(in_features=784, out_features=10, bias=True)
)<jupyter_text>4. Train the modelNow it's time to train the model. We will train it for 100 *epochs* (iterations) with a *batch size* of 2048 (the number of training examples to evaluate prior to doing gradient descent), and record the training and validation losses while training.<jupyter_code># Use GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"

# Moving model to use GPU
model.to(device)
def getPredsFromLogits(logits):
  # Using softmax to get an array that sums to 1, and then getting the index with the highest value
  return torch.nn.functional.softmax(logits, dim=1).argmax(dim=1)

epochs = 100
train_losses = []
train_accuracies = []
for epoch in range(1, epochs+1):
    train_loss = 0.0

    train_counts = 0

    ###################
    # train the model #
    ###################

    # Setting model to train mode
    model.train()

    for images, labels in training_dataloader:

        # Moving data to GPU if available
        images, labels = images.to(device), labels.to(device)
        
        # Setting all gradients to zero
        optimizer.zero_grad()

        # Calculate Output
        output = model(images)
            
        # Calculate Loss
        loss = criterion(output, labels)

        # Calculate Gradients
        loss.backward()

        # Perform Gradient Descent Step
        optimizer.step()

        # Saving loss
        train_loss += loss.item()

        # Get Predictions
        train_preds = getPredsFromLogits(output)

        # Saving number of right predictions for accuracy
        train_counts += train_preds.eq(labels).sum().item()

    # Averaging and Saving Losses
    train_loss/=len(training_data)
    train_losses.append(train_loss)

    # Getting accuracies and saving them
    train_acc = train_counts/len(training_data)
    train_accuracies.append(train_acc)


    print('Epoch: {} \tTraining Loss: {:.6f} \tTraining Accuracy: {:.2f}%'.format(epoch, train_loss, train_acc*100))
plt.plot(train_losses)
plt.xlabel('epoch')
plt.ylabel('Mean Squared Error')
plt.title('Training Loss')
plt.show()<jupyter_output><empty_output><jupyter_text>As the above plot suggests, our model has not converged to the optimal parameters yet after training for 50 epochs (iterations). This suggests the accuracy may improve with more training (more epochs). Further, the validation accuracy peaks around ~93%.    5. Evaluate the modelNow that we trained our model, it's time to evaluate it using the test dataset, which we did not use when training the model. This gives us a sense of how well our model predicts unseen data, which is the case when we use it in the real world.<jupyter_code>test_loss = 0.0
test_counts = 0

# Setting model to evaluation mode, no parameters will change
model.eval()

for images, labels in test_dataloader:

    # Moving to GPU if available
    images, labels = images.to(device), labels.to(device)

    # Calculate Output
    output = model(images)

    # Calculate Loss
    loss = criterion(output, labels)

    # Saving loss
    test_loss += loss.item()

    # Get Predictions
    test_preds = getPredsFromLogits(output)

    # Saving number of right predictions for accuracy
    test_counts += test_preds.eq(labels).sum().item()

# Calculating test accuracy
test_acc = test_counts/len(test_data)
print('Test Loss: {:.6f} \tTest Accuracy: {:.2f}%'.format(test_loss, test_acc*100))
# Plotting Training label distribution
plt.hist(training_data.targets)
plt.xlabel('Classes')
plt.ylabel('Frequency')
plt.title('Training Labels Distribution');
# Plotting testing label distribution
plt.hist(test_data.targets)
plt.xlabel('Classes')
plt.ylabel('Frequency')
plt.title('Test Labels Distribution');<jupyter_output><empty_output><jupyter_text>Wow! Our softmax classifier fit the MNIST digit data pretty well, correctly predicting the unseen handwritten digits around 88% to 89% of the time. Since the dataset is fairly balanced (i.e., the digit classes are close to equally represented), we can be relatively confident that the test set accuracy score provides a good estimate for how the model will perform on similar unseen handwritten digit data.  Summary- We use *one-hot encoding* to represent categorical data.- Softmax regression is probably the popular classification technique and it is a foundational algorithm for classification methods in deep learning (neural networks). - The *softmax* logit function maps the input features to a probability distribution that guarantees the values are nonnegative and sum up to 1.- Softmax regression classifiers try to *maximize the likelihood* of the observed data, like logistic regression.- Categorical *cross-entorpy* is the loss function for softmax regression.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07755775577557755, 'fraction_numerical': 0.013601360136013601, 'mean_word_length': 4.0643200810331725, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 25, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '601523', 'license': 'MIT', 'n_tokens_mistral': 5518, 'n_tokens_neox': 5133, 'n_words': 2514, 'path': 'machine_learning/lesson 2 - logistic regression/Classification_Softmax_Regression.ipynb', 'repo_name': 'BreakoutMentors/Data-Science-and-Machine-Learning', 'size': '19998'}