<jupyter_start><jupyter_text>Implementation of Softmax Regression from Scratch:label:`chapter_softmax_scratch`Just as we implemented linear regression from scratch,we believe that multiclass logistic (softmax) regressionis similarly fundamental and you ought to knowthe gory details of how to implement it from scratch.As with linear regression, after doing things by handwe will breeze through an implementation in Gluon for comparison.To begin, let's import our packages.<jupyter_code>import sys
sys.path.insert(0, '..')

%matplotlib inline
import d2l
import torch
from torch.distributions import normal<jupyter_output><empty_output><jupyter_text>We will work with the Fashion-MNIST dataset just introduced,cuing up an iterator with batch size 256.<jupyter_code>batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)<jupyter_output><empty_output><jupyter_text>Initialize Model ParametersJust as in linear regression, we represent each example as a vector.Since each example is a $28 \times 28$ image,we can flatten each example, treating them as $784$ dimensional vectors.In the future, we'll talk about more sophisticated strategiesfor exploiting the spatial structure in images,but for now we treat each pixel location as just another feature.Recall that in softmax regression,we have as many outputs as there are categories.Because our dataset has $10$ categories,our network will have an output dimension of $10$.Consequently, our weights will constitute a $784 \times 10$ matrixand the biases will constitute a $1 \times 10$ vector.As with linear regression, we will initialize our weights $W$with Gaussian noise and our biases to take the initial value $0$.<jupyter_code>num_inputs = 784
num_outputs = 10

W = normal.Normal(loc = 0, scale = 0.01).sample((num_inputs, num_outputs))
b = torch.zeros(num_outputs)<jupyter_output><empty_output><jupyter_text>Recall that we need to *attach gradients* to the model parameters.More literally, we are allocating memory for future gradients to be storedand notifiying PyTorch that we want gradients to be calculated with respect to these parameters in the first place.<jupyter_code>W.requires_grad_(True)
b.requires_grad_(True)<jupyter_output><empty_output><jupyter_text>The SoftmaxBefore implementing the softmax regression model,let's briefly review how `torch.sum` workalong specific dimensions in a PyTorch tensor.Given a matrix `X` we can sum over all elements (default) or onlyover elements in the same column (`dim=0`) or the same row (`dim=1`).Note that if `X` is an array with shape `(2, 3)`and we sum over the columns (`torch.sum(X, dim=0`),the result will be a (1D) vector with shape `(3,)`.If we want to keep the number of axes in the original array(resulting in a 2D array with shape `(1,3)`),rather than collapsing out the dimension that we summed overwe can specify `keepdim=True` when invoking `torch.sum`.<jupyter_code>X = torch.tensor([[1, 2, 3], [4, 5, 6]])
torch.sum(X, dim=0, keepdim=True), torch.sum(X, dim=1, keepdim=True)<jupyter_output><empty_output><jupyter_text>We are now ready to implement the softmax function.Recall that softmax consists of two steps:First, we exponentiate each term (using `torch.exp`).Then, we sum over each row (we have one row per example in the batch)to get the normalization constants for each example.Finally, we divide each row by its normalization constant,ensuring that the result sums to $1$.Before looking at the code, let's recallwhat this looks expressed as an equation:$$\mathrm{softmax}(\mathbf{X})_{ij} = \frac{\exp(X_{ij})}{\sum_k \exp(X_{ik})}$$The denominator, or normalization constant,is also sometimes called the partition function(and its logarithm the log-partition function).The origins of that name are in [statistical physics](https://en.wikipedia.org/wiki/Partition_function_(statistical_mechanics))where a related equation models the distributionover an ensemble of particles).<jupyter_code>def softmax(X):
    X_exp = torch.exp(X)
    partition = torch.sum(X_exp, dim=1, keepdim=True)
    return X_exp / partition  # The broadcast mechanism is applied here<jupyter_output><empty_output><jupyter_text>As you can see, for any random input, we turn each element into a non-negative number. Moreover, each row sums up to 1, as is required for a probability.Note that while this looks correct mathematically,we were a bit sloppy in our implementationbecause failed to take precautions against numerical overflow or underflowdue to large (or very small) elements of the matrix,as we did in:numref:`chapter_naive_bayes`.<jupyter_code># X = nd.random.normal(shape=(2, 5))
X = normal.Normal(loc = 0, scale = 1).sample((2, 5))
X_prob = softmax(X)
X_prob, torch.sum(X_prob, dim=1)<jupyter_output><empty_output><jupyter_text>The ModelNow that we have defined the softmax operation,we can implement the softmax regression model.The below code defines the forward pass through the network.Note that we flatten each original image in the batchinto a vector with length `num_inputs` with the `view` functionbefore passing the data through our model.<jupyter_code>def net(X):
    return softmax(torch.matmul(X.reshape((-1, num_inputs)), W) + b)<jupyter_output><empty_output><jupyter_text>The Loss FunctionNext, we need to implement the cross entropy loss function,introduced in :numref:`chapter_softmax`.This may be the most common loss functionin all of deep learning because, at the moment,classification problems far outnumber regression problems.Recall that cross entropy takes the negative log likelihoodof the predicted probability assigned to the true label $-\log p(y|x)$.Rather than iterating over the predictions with a Python `for` loop(which tends to be inefficient), we can use the `gather` functionwhich allows us to select the appropriate termsfrom the matrix of softmax entries easily.Below, we illustrate the `gather` function on a toy example,with 3 categories and 2 examples.<jupyter_code>y_hat = torch.tensor([[0.1, 0.3, 0.6], [0.3, 0.2, 0.5]])
y = torch.tensor([0, 2])
torch.gather(y_hat, 1, y.unsqueeze(dim=1)) # y has to be unsqueezed so that shape(y_hat) = shape(y)<jupyter_output><empty_output><jupyter_text>Now we can implement the cross-entropy loss function efficientlywith just one line of code.<jupyter_code>def cross_entropy(y_hat, y):
    return -torch.gather(y_hat, 1, y.unsqueeze(dim=1)).log()<jupyter_output><empty_output><jupyter_text>Classification AccuracyGiven the predicted probability distribution `y_hat`,we typically choose the class with highest predicted probabilitywhenever we must output a *hard* prediction. Indeed, many applications require that we make a choice. Gmail must catetegorize an email into Primary, Social, Updates, or Forums. It might estimate probabilities internally, but at the end of the day it has to choose one among the categories.When predictions are consistent with the actual category `y`, they are correct. The classification accuracy is the fraction of all predictions that are correct. Although we cannot optimize accuracy directly (it is not differentiable), it's often the performance metric that we care most about, and we will nearly always report it when training classifiers.To compute accuracy we do the following:First, we execute `y_hat.argmax(dim=1)`to gather the predicted classes(given by the indices for the largest entires each row).The result has the same shape as the variable `y`.Now we just need to check how frequently the two match. The result is PyTorch tensor containing entries of 0 (false) and 1 (true). Since the attribute `mean` can only calculate the mean of floating types,we also need to convert the result to `float`. Taking the mean yields the desired result.<jupyter_code>def accuracy(y_hat, y):
    return (y_hat.argmax(dim=1) == y).float().mean().item()<jupyter_output><empty_output><jupyter_text>We will continue to use the variables `y_hat` and `y`defined in the `gather` function,as the predicted probability distribution and label, respectively.We can see that the first example's prediction category is 2(the largest element of the row is 0.6 with an index of 2),which is inconsistent with the actual label, 0.The second example's prediction category is 2(the largest element of the row is 0.5 with an index of 2),which is consistent with the actual label, 2.Therefore, the classification accuracy rate for these two examples is 0.5.<jupyter_code>accuracy(y_hat, y)<jupyter_output><empty_output><jupyter_text>Similarly, we can evaluate the accuracy for model `net` on the data set(accessed via `data_iter`).<jupyter_code># The function will be gradually improved: the complete implementation will be
# discussed in the "Image Augmentation" section
def evaluate_accuracy(data_iter, net):
    acc_sum, n = 0.0, 0
    for X, y in data_iter:
        acc_sum += (net(X).argmax(dim=1) == y).sum().item()
        n += y.size()[0]  # y.size()[0] = batch_size
    return acc_sum / n<jupyter_output><empty_output><jupyter_text>Because we initialized the `net` model with random weights,the accuracy of this model should be close to random guessing,i.e. 0.1 for 10 classes.<jupyter_code>evaluate_accuracy(test_iter, net)<jupyter_output><empty_output><jupyter_text>Model TrainingThe training loop for softmax regression should look strikingly familiarif you read through our implementationof linear regression earlier in this chapter.Again, we use the mini-batch stochastic gradient descentto optimize the loss function of the model.Note that the number of epochs (`num_epochs`),and learning rate (`lr`) are both adjustable hyper-parameters.By changing their values, we may be able to increase the classification accuracy of the model. In practice we'll want to split our data three waysinto training, validation, and test data, using the validation data to choose the best values of our hyperparameters.<jupyter_code>num_epochs, lr = 5, 0.1

# This function has been saved in the d2l package for future use
def train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, params=None, lr=None, trainer=None):
    for epoch in range(num_epochs):
        train_l_sum, train_acc_sum, n = 0.0, 0.0, 0
        for X, y in train_iter:
            y_hat = net(X)
            l = loss(y_hat, y).sum()
            l.backward()
            if trainer is None:
                d2l.sgd(params, lr, batch_size)
            else:
                # This will be illustrated in the next section
                trainer.step(batch_size)
            train_l_sum += l.item()
            train_acc_sum += (y_hat.argmax(dim=1) == y).sum().item()
            n += y.size()[0]
        test_acc = evaluate_accuracy(test_iter, net)
        print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f'
              % (epoch + 1, train_l_sum / n, train_acc_sum / n, test_acc))

train_ch3(net, train_iter, test_iter, cross_entropy, num_epochs, batch_size, [W, b], lr)<jupyter_output>epoch 1, loss 0.7884, train acc 0.747, test acc 0.794
epoch 2, loss 0.5722, train acc 0.812, test acc 0.804
epoch 3, loss 0.5248, train acc 0.825, test acc 0.817
epoch 4, loss 0.5007, train acc 0.833, test acc 0.826
epoch 5, loss 0.4853, train acc 0.837, test acc 0.825<jupyter_text>PredictionNow that training is complete, our model is ready to classify some images.Given a series of images, we will compare their actual labels(first line of text output) and the model predictions(second line of text output).<jupyter_code>for X, y in test_iter:
    break

true_labels = d2l.get_fashion_mnist_labels(y.numpy())
pred_labels = d2l.get_fashion_mnist_labels(net(X).argmax(dim=1).numpy())
titles = [truelabel + '\n' + predlabel for truelabel, predlabel in zip(true_labels, pred_labels)]

d2l.show_fashion_mnist(X[10:20], titles[10:20])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08239095315024232, 'fraction_numerical': 0.01811070487203469, 'mean_word_length': 5.174278215223097, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 64, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '10159', 'license': 'Apache-2.0', 'n_tokens_mistral': 3563, 'n_tokens_neox': 3352, 'n_words': 1409, 'path': 'Ch05_Linear_Neural_Networks/Implementation_of_Softmax_Regression_from_Scratch.ipynb', 'repo_name': 'milk-ways/d2l-pytorch', 'size': '11761'}