<jupyter_start><jupyter_text>Implementation of Multilayer Perceptron from Scratch:label:`chapter_mlp_scratch`Now that we know how multilayer perceptrons (MLPs) work in theory,let's implement them. First, we import the required packages.<jupyter_code>import d2l
from mxnet import nd, gluon<jupyter_output><empty_output><jupyter_text>To compare against the resultswe previously achieved with vanilla softmax regression,we continue to use the Fashion-MNIST image classification dataset.<jupyter_code>batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)<jupyter_output><empty_output><jupyter_text>Initialize Model ParametersRecall that this dataset contains 10 classes and thateach image consists of a $28 \times 28 = 784$ grid of pixel values.Since we'll be discarding the spatial strucutre (for now),we can just think of this as a classifiation datasetwith $784$ input features and $10$ classes.In particular we will implement our MLPwith one hidden layer and $256$ hidden units.Note that we can regard both of these choices as *hyperparameters*that could be set based on performance on validation data.Typically, we'll choose layer widths as powers of $2$to make everything align nicely in memory.Again, we will allocate several NDArrays to represent our parameters.Note that we now have one weight matrix and one bias vector *per layer*.As always, we must call `attach_grad` to allocate memory for the gradients with respect to these parameters.<jupyter_code>num_inputs, num_outputs, num_hiddens = 784, 10, 256

W1 = nd.random.normal(scale=0.01, shape=(num_inputs, num_hiddens))
b1 = nd.zeros(num_hiddens)
W2 = nd.random.normal(scale=0.01, shape=(num_hiddens, num_outputs))
b2 = nd.zeros(num_outputs)
params = [W1, b1, W2, b2]

for param in params:
    param.attach_grad()<jupyter_output><empty_output><jupyter_text>Activation FunctionTo make sure we know how everything works,we will use the `maximum` function to implement ReLU ourselves,instead of invoking `nd.relu` directly.<jupyter_code>def relu(X):
    return nd.maximum(X, 0)<jupyter_output><empty_output><jupyter_text>The modelAs in softmax regression, we will `reshape` each 2D imageinto a flat vector of length  `num_inputs`.Finally, we can implement our model with just a few lines of code.<jupyter_code>def net(X):
    X = X.reshape((-1, num_inputs))
    H = relu(nd.dot(X, W1) + b1)
    return nd.dot(H, W2) + b2<jupyter_output><empty_output><jupyter_text>The Loss FunctionFor better numerical stability and because we already knowhow to implement softmax regression completely from scratch in :numref:`chapter_softmax_scratch`,we will use Gluon's integrated functionfor calculating the softmax and cross-entropy loss.Recall that we discussed some of these intricaciesin :numref:`chapter_mlp`.We encourage the interested reader to examing the source codefor `mxnet.gluon.loss.SoftmaxCrossEntropyLoss` for more details.<jupyter_code>loss = gluon.loss.SoftmaxCrossEntropyLoss()<jupyter_output><empty_output><jupyter_text>TrainingSteps for training the MLP are no different than for softmax regression.In the `d2l` package, we directly call the `train_ch3` function, whose implementation was introduced in :numref:`chapter_softmax_scratch`.We set the number of epochs to $10$ and the learning rate to $0.5$.<jupyter_code>num_epochs, lr = 10, 0.5
d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, 
             lambda batch_size: d2l.sgd(params, lr, batch_size))<jupyter_output><empty_output><jupyter_text>To see how well we did, let's apply the model to some test data.If you're interested, compare the result to corresponding linear model in :numref:`chapter_softmax_scratch`.<jupyter_code>d2l.predict_ch3(net, test_iter)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07704612103439083, 'fraction_numerical': 0.017861903492402027, 'mean_word_length': 6.42970297029703, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '52115', 'license': 'MIT', 'n_tokens_mistral': 1214, 'n_tokens_neox': 1141, 'n_words': 402, 'path': 'd2l-en/chapter_multilayer-perceptrons/mlp-scratch.ipynb', 'repo_name': 'mru4913/Dive-into-Deep-Learning', 'size': '3751'}