<jupyter_start><jupyter_text>Residual Networks (ResNet):label:`sec_resnet`As we design increasingly deeper networks it becomes imperative to understand how adding layers can increase the complexity and expressiveness of the network.Even more important is the ability to design networks where adding layers makes networks strictly more expressive rather than just different.To make some progress we need a bit of mathematics. Function ClassesConsider $\mathcal{F}$, the class of functions that a specific network architecture (together with learning rates and other hyperparameter settings) can reach.That is, for all $f \in \mathcal{F}$ there exists some set of parameters (e.g., weights and biases) that can be obtained through training on a suitable dataset.Let us assume that $f^*$ is the "truth" function that we really would like to find.If it is in $\mathcal{F}$, we are in good shape but typically we will not be quite so lucky.Instead, we will try to find some $f^*_\mathcal{F}$ which is our best bet within $\mathcal{F}$.For instance, given a dataset with features $\mathbf{X}$and labels $\mathbf{y}$,we might try finding it by solving the following optimization problem:$$f^*_\mathcal{F} \stackrel{\mathrm{def}}{=} \mathop{\mathrm{argmin}}_f L(\mathbf{X}, \mathbf{y}, f) \text{ subject to } f \in \mathcal{F}.$$It is only reasonable to assume that if we design a different and more powerful architecture $\mathcal{F}'$ we should arrive at a better outcome. In other words, we would expect that $f^*_{\mathcal{F}'}$ is "better" than $f^*_{\mathcal{F}}$. However, if $\mathcal{F} \not\subseteq \mathcal{F}'$ there is no guarantee that this should even happen. In fact, $f^*_{\mathcal{F}'}$ might well be worse. As illustrated by :numref:`fig_functionclasses`,for non-nested function classes, a larger function class does not always move closer to the "truth" function $f^*$. For instance,on the left of :numref:`fig_functionclasses`,though $\mathcal{F}_3$ is closer to $f^*$ than $\mathcal{F}_1$, $\mathcal{F}_6$ moves away and there is no guarantee that further increasing the complexity can reduce the distance from $f^*$.With nested function classeswhere $\mathcal{F}_1 \subseteq \ldots \subseteq \mathcal{F}_6$on the right of :numref:`fig_functionclasses`,we can avoid the aforementioned issue from the non-nested function classes.![For non-nested function classes, a larger (indicated by area) function class does not guarantee to get closer to the "truth" function ($f^*$). This does not happen in nested function classes.](../img/functionclasses.svg):label:`fig_functionclasses`Thus,only if larger function classes contain the smaller ones are we guaranteed that increasing them strictly increases the expressive power of the network.For deep neural networks,if we can train the newly-added layer into an identity function $f(\mathbf{x}) = \mathbf{x}$, the new model will be as effective as the original model. As the new model may get a better solution to fit the training dataset, the added layer might make it easier to reduce training errors.This is the question that He et al. considered when working on very deep computer vision models :cite:`He.Zhang.Ren.ea.2016`. At the heart of their proposed *residual network* (*ResNet*) is the idea that every additional layer should more easilycontain the identity function as one of its elements. These considerations are rather profound but they led to a surprisingly simplesolution, a *residual block*.With it, ResNet won the ImageNet Large Scale Visual Recognition Challenge in 2015. The design had a profound influence on how tobuild deep neural networks. Residual BlocksLet us focus on a local part of a neural network, as depicted in :numref:`fig_residual_block`. Denote the input by $\mathbf{x}$.We assume that the desired underlying mapping we want to obtain by learning is $f(\mathbf{x})$, to be used as the input to the activation function on the top.On the left of :numref:`fig_residual_block`,the portion within the dotted-line box must directly learn the mapping $f(\mathbf{x})$.On the right,the portion within the dotted-line boxneeds tolearn the *residual mapping* $f(\mathbf{x}) - \mathbf{x}$,which is how the residual block derives its name.If the identity mapping $f(\mathbf{x}) = \mathbf{x}$ is the desired underlying mapping,the residual mapping is easier to learn:we only need to push the weights and biasesof theupper weight layer (e.g., fully-connected layer and convolutional layer)within the dotted-line boxto zero.The right figure in :numref:`fig_residual_block` illustrates the  *residual block* of ResNet,where the solid line carrying the layer input $\mathbf{x}$ to the addition operatoris called a *residual connection* (or *shortcut connection*).With residual blocks, inputs can forward propagate faster through the residual connections across layers.![A regular block (left) and a residual block (right).](../img/residual-block.svg):label:`fig_residual_block`ResNet follows VGG's full $3\times 3$ convolutional layer design. The residual block has two $3\times 3$ convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Then, we skip these two convolution operations and add the input directly before the final ReLU activation function.This kind of design requires that the output of the two convolutional layers has to be of the same shape as the input, so that they can be added together. If we want to change the number of channels, we need to introduce an additional $1\times 1$ convolutional layer to transform the input into the desired shape for the addition operation. Let us have a look at the code below.<jupyter_code>from d2l import mxnet as d2l
from mxnet import np, npx
from mxnet.gluon import nn
npx.set_np()

class Residual(nn.Block):  #@save
    """The Residual block of ResNet."""
    def __init__(self, num_channels, use_1x1conv=False, strides=1, **kwargs):
        super().__init__(**kwargs)
        self.conv1 = nn.Conv2D(num_channels, kernel_size=3, padding=1,
                               strides=strides)
        self.conv2 = nn.Conv2D(num_channels, kernel_size=3, padding=1)
        if use_1x1conv:
            self.conv3 = nn.Conv2D(num_channels, kernel_size=1,
                                   strides=strides)
        else:
            self.conv3 = None
        self.bn1 = nn.BatchNorm()
        self.bn2 = nn.BatchNorm()

    def forward(self, X):
        Y = npx.relu(self.bn1(self.conv1(X)))
        Y = self.bn2(self.conv2(Y))
        if self.conv3:
            X = self.conv3(X)
        return npx.relu(Y + X)<jupyter_output><empty_output><jupyter_text>This code generates two types of networks: one where we add the input to the output before applying the ReLU nonlinearity whenever `use_1x1conv=False`, and one where we adjust channels and resolution by means of a $1 \times 1$ convolution before adding. :numref:`fig_resnet_block` illustrates this:![ResNet block with and without $1 \times 1$ convolution.](../img/resnet-block.svg):label:`fig_resnet_block`Now let us look at a situation where the input and output are of the same shape.<jupyter_code>blk = Residual(3)
blk.initialize()
X = np.random.uniform(size=(4, 3, 6, 6))
blk(X).shape<jupyter_output><empty_output><jupyter_text>We also have the option to halve the output height and width while increasing the number of output channels.<jupyter_code>blk = Residual(6, use_1x1conv=True, strides=2)
blk.initialize()
blk(X).shape<jupyter_output><empty_output><jupyter_text>ResNet ModelThe first two layers of ResNet are the same as those of the GoogLeNet we described before: the $7\times 7$ convolutional layer with 64 output channels and a stride of 2 is followed by the $3\times 3$ maximum pooling layer with a stride of 2. The difference is the batch normalization layer added after each convolutional layer in ResNet.<jupyter_code>net = nn.Sequential()
net.add(nn.Conv2D(64, kernel_size=7, strides=2, padding=3),
        nn.BatchNorm(), nn.Activation('relu'),
        nn.MaxPool2D(pool_size=3, strides=2, padding=1))<jupyter_output><empty_output><jupyter_text>GoogLeNet uses four modules made up of Inception blocks.However, ResNet uses four modules made up of residual blocks, each of which uses several residual blocks with the same number of output channels. The number of channels in the first module is the same as the number of input channels. Since a maximum pooling layer with a stride of 2 has already been used, it is not necessary to reduce the height and width. In the first residual block for each of the subsequent modules, the number of channels is doubled compared with that of the previous module, and the height and width are halved.Now, we implement this module. Note that special processing has been performed on the first module.<jupyter_code>def resnet_block(num_channels, num_residuals, first_block=False):
    blk = nn.Sequential()
    for i in range(num_residuals):
        if i == 0 and not first_block:
            blk.add(Residual(num_channels, use_1x1conv=True, strides=2))
        else:
            blk.add(Residual(num_channels))
    return blk<jupyter_output><empty_output><jupyter_text>Then, we add all the modules to ResNet. Here, two residual blocks are used for each module.<jupyter_code>net.add(resnet_block(64, 2, first_block=True),
        resnet_block(128, 2),
        resnet_block(256, 2),
        resnet_block(512, 2))<jupyter_output><empty_output><jupyter_text>Finally, just like GoogLeNet, we add a global average pooling layer, followed by the fully-connected layer output.<jupyter_code>net.add(nn.GlobalAvgPool2D(), nn.Dense(10))<jupyter_output><empty_output><jupyter_text>There are 4 convolutional layers in each module (excluding the $1\times 1$ convolutional layer). Together with the first $7\times 7$ convolutional layer and the final fully-connected layer, there are 18 layers in total. Therefore, this model is commonly known as ResNet-18.By configuring different numbers of channels and residual blocks in the module, we can create different ResNet models, such as the deeper 152-layer ResNet-152. Although the main architecture of ResNet is similar to that of GoogLeNet, ResNet's structure is simpler and easier to modify. All these factors have resulted in the rapid and widespread use of ResNet. :numref:`fig_resnet18` depicts the full ResNet-18.![The ResNet-18 architecture.](../img/resnet18.svg):label:`fig_resnet18`Before training ResNet, let us observe how the input shape changes across different modules in ResNet. As in all the previous architectures, the resolution decreases while the number of channels increases up until the point where a global average pooling layer aggregates all features.<jupyter_code>X = np.random.uniform(size=(1, 1, 224, 224))
net.initialize()
for layer in net:
    X = layer(X)
    print(layer.name, 'output shape:\t', X.shape)<jupyter_output>conv5 output shape:	 (1, 64, 112, 112)
batchnorm4 output shape:	 (1, 64, 112, 112)
relu0 output shape:	 (1, 64, 112, 112)
pool0 output shape:	 (1, 64, 56, 56)
sequential1 output shape:	 (1, 64, 56, 56)
sequential2 output shape:	 (1, 128, 28, 28)
sequential3 output shape:	 (1, 256, 14, 14)
sequential4 output shape:	 (1, 512, 7, 7)
pool1 output shape:	 (1, 512, 1, 1)
dense0 output shape:	 (1, 10)<jupyter_text>TrainingWe train ResNet on the Fashion-MNIST dataset, just like before.<jupyter_code>lr, num_epochs, batch_size = 0.05, 10, 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size, resize=96)
d2l.train_ch6(net, train_iter, test_iter, num_epochs, lr)<jupyter_output>loss 0.012, train acc 0.997, test acc 0.881
4852.1 examples/sec on gpu(0)
{'fraction_non_alphanumeric': 0.08381799521040027, 'fraction_numerical': 0.021211084502223743, 'mean_word_length': 4.745945945945946, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '8168', 'license': 'MIT', 'n_tokens_mistral': 3442, 'n_tokens_neox': 3190, 'n_words': 1468, 'path': 'd2l-en/mxnet/chapter_convolutional-modern/resnet.ipynb', 'repo_name': 'gr8khan/d2lai', 'size': '11692'}