<jupyter_start><jupyter_text>Concise Implementation of Softmax Regression:label:`sec_softmax_concise`Just as high-level APIs of deep learning frameworksmade it much easierto implement linear regression in :numref:`sec_linear_concise`,we will find it similarly (or possibly more)convenient for implementing classification models. Let us stick with the Fashion-MNIST datasetand keep the batch size at 256 as in :numref:`sec_softmax_scratch`.<jupyter_code>from d2l import mxnet as d2l
from mxnet import gluon, init, npx
from mxnet.gluon import nn
npx.set_np()
batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)<jupyter_output><empty_output><jupyter_text>Initializing Model ParametersAs mentioned in :numref:`sec_softmax`,the output layer of softmax regressionis a fully-connected layer.Therefore, to implement our model,we just need to add one fully-connected layerwith 10 outputs to our `Sequential`.Again, here, the `Sequential` is not really necessary,but we might as well form the habit since it will be ubiquitouswhen implementing deep models.Again, we initialize the weights at randomwith zero mean and standard deviation 0.01.<jupyter_code>net = nn.Sequential()
net.add(nn.Dense(10))
net.initialize(init.Normal(sigma=0.01))<jupyter_output><empty_output><jupyter_text>Softmax Implementation Revisited:label:`subsec_softmax-implementation-revisited`In the previous example of :numref:`sec_softmax_scratch`,we calculated our model's outputand then ran this output through the cross-entropy loss.Mathematically, that is a perfectly reasonable thing to do.However, from a computational perspective,exponentiation can be a source of numerical stability issues.Recall that the softmax function calculates$\hat y_j = \frac{\exp(o_j)}{\sum_k \exp(o_k)}$,where $\hat y_j$ is the $j^\mathrm{th}$ element ofthe predicted probability distribution $\hat{\mathbf{y}}$and $o_j$ is the $j^\mathrm{th}$ element of the logits$\mathbf{o}$.If some of the $o_k$ are very large (i.e., very positive),then $\exp(o_k)$ might be larger than the largest numberwe can have for certain data types (i.e., *overflow*).This would make the denominator (and/or numerator) `inf` (infinity)and we wind up encountering either 0, `inf`, or `nan` (not a number) for $\hat y_j$.In these situations we do not get a well-definedreturn value for cross-entropy.One trick to get around this is to first subtract $\max(o_k)$from all $o_k$ before proceeding with the softmax calculation.You can verify that this shifting of each $o_k$ by constant factordoes not change the return value of softmax.After the subtraction and normalization step,it might be possible that some $o_j$ have large negative valuesand thus that the corresponding $\exp(o_j)$ will take values close to zero.These might be rounded to zero due to finite precision (i.e., *underflow*),making $\hat y_j$ zero and giving us `-inf` for $\log(\hat y_j)$.A few steps down the road in backpropagation,we might find ourselves faced with a screenfulof the dreaded `nan` results.Fortunately, we are saved by the fact thateven though we are computing exponential functions,we ultimately intend to take their log(when calculating the cross-entropy loss).By combining these two operatorssoftmax and cross-entropy together,we can escape the numerical stability issuesthat might otherwise plague us during backpropagation.As shown in the equation below, we avoid calculating $\exp(o_j)$and can use instead $o_j$ directly due to the canceling in $\log(\exp(\cdot))$.$$\begin{aligned}\log{(\hat y_j)} & = \log\left( \frac{\exp(o_j)}{\sum_k \exp(o_k)}\right) \\& = \log{(\exp(o_j))}-\log{\left( \sum_k \exp(o_k) \right)} \\& = o_j -\log{\left( \sum_k \exp(o_k) \right)}.\end{aligned}$$We will want to keep the conventional softmax function handyin case we ever want to evaluate the output probabilities by our model.But instead of passing softmax probabilities into our new loss function,we will just pass the logits and compute the softmax and its logall at once inside the cross-entropy loss function,which does smart things like the ["LogSumExp trick"](https://en.wikipedia.org/wiki/LogSumExp).<jupyter_code>loss = gluon.loss.SoftmaxCrossEntropyLoss()<jupyter_output><empty_output><jupyter_text>Optimization AlgorithmHere, we use minibatch stochastic gradient descentwith a learning rate of 0.1 as the optimization algorithm.Note that this is the same as we applied in the linear regression exampleand it illustrates the general applicability of the optimizers.<jupyter_code>trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.1})<jupyter_output><empty_output><jupyter_text>TrainingNext we call the training function defined in :numref:`sec_softmax_scratch` to train the model.<jupyter_code>num_epochs = 10
d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08810844115834873, 'fraction_numerical': 0.005750667488190594, 'mean_word_length': 6.078488372093023, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '9639', 'license': 'MIT', 'n_tokens_mistral': 1439, 'n_tokens_neox': 1367, 'n_words': 554, 'path': 'd2l-en/mxnet/chapter_linear-networks/softmax-regression-concise.ipynb', 'repo_name': 'gr8khan/d2lai', 'size': '4869'}