<jupyter_start><jupyter_text>Network in Network (NiN):label:`chapter_nin`LeNet, AlexNet, and VGG all share a common design pattern:extract features exploiting *spatial* structurevia a sequence of convolutions and pooling layersand then post-process the representations via fully-connected layers.The improvements upon LeNet by AlexNet and VGG mainly liein how these later networks widen and deepen these two modules.Alternatively, one could imagine using fully-connected layersearlier in the process.However, a careless use of dense layers might give up thespatial structure of the representation entirely,Network in Network (NiN) blocks offer an alternative.They were proposed by [<NAME> and Yan, 2013](https://arxiv.org/pdf/1312.4400.pdf) based on a very simple insight---touse an MLP on the channels for each pixel separately. NiN BlocksRecall that the inputs and outputs of convolutional layersconsist of four-dimensional arrays with axescorresponding to the batch, channel, height, and width.Also recall that the inputs and outputs of fully-connected layersare typically two-dimensional arrays corresponding to the batch, and features.The idea behind NiN is to apply a fully-connected layerat each pixel location (for each height and  width).If we tie the weights across each spatial location,we could think of this as a $1\times 1$ convolutional layer(as described in :numref:`chapter_channels`)or as a fully-connected layer acting independently on each pixel location.Another way to view this is to think of each element in the spatial dimension(height and width) as equivalent to an exampleand the channel as equivalent to a feature.The figure below illustrates the main structural differencesbetween NiN and AlexNet, VGG, and other networks.<jupyter_code>from IPython.display import SVG
SVG(filename='../img/nin-compare.svg')<jupyter_output><empty_output><jupyter_text>![The figure on the left shows the network structure of AlexNet and VGG, and the figure on the right shows the network structure of NiN. ]The NiN block consists of one convolutional layerfollowed by two $1\times 1$ convolutional layers that act asper-pixel fully-connected layers with ReLU activations.The convolution width of the first layer is typically set by the user.The subsequent widths are fixed to $1 \times 1$.<jupyter_code>import sys
sys.path.insert(0, '..')

import d2l
import torch
import torch.nn as nn

def nin_block(in_channels,out_channels,kernel_size,strides,padding):
    blk = nn.Sequential(
            nn.Conv2d(in_channels,out_channels,kernel_size,strides,padding),
            nn.ReLU(),
            nn.Conv2d(out_channels,out_channels,kernel_size=1),
            nn.ReLU(),
            nn.Conv2d(out_channels,out_channels,kernel_size=1),
            nn.ReLU())
    return blk<jupyter_output><empty_output><jupyter_text>NiN ModelThe original NiN network was proposed shortly after AlexNetand clearly draws some inspiration.NiN uses convolutional layers with window shapesof $11\times 11$, $5\times 5$, and $3\times 3$,and the corresponding numbers of output channels are the same as in AlexNet. Each NiN block is followed by a maximum pooling layerwith a stride of 2 and a window shape of $3\times 3$.Once significant difference between NiN and AlexNetis that NiN avoids dense connections altogether.Instead, NiN uses an NiN block with a number of output channels equal to the number of label classes, followed by a *global* average pooling layer,yielding a vector of logits.One advantage of NiN's design is that it significantlyreduces the number of required model parameters.However, in practice, this design sometimes requiresincreased model training time.<jupyter_code>class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.n1 = nin_block(1,out_channels=96, kernel_size=11, strides=4, padding=0)
        self.m1 = nn.MaxPool2d(3,stride=2)
        self.n2 = nin_block(96,out_channels=256, kernel_size=5, strides=1, padding=2)
        self.m2 = nn.MaxPool2d(3,stride=2)
        self.n3 = nin_block(256,out_channels=384, kernel_size=3, strides=1, padding=1)
        self.m3 = nn.MaxPool2d(3,stride=2)
        self.dropout1 = nn.Dropout2d(0.5)
        self.n4 = nin_block(384,out_channels=10, kernel_size=3, strides=1, padding=1)
        #Global Average Pooling can be achieved by AdaptiveMaxPool2d with output size = (1,1)
        self.avg1 = nn.AdaptiveMaxPool2d((1,1))
        self.flat = Flatten()
        
    def forward(self, x): 
        x = self.m1(self.n1(x))
        x = self.m2(self.n2(x))
        x = self.dropout1(self.m3(self.n3(x)))
        x = self.n4(x)
        x = self.avg1(x)
        x = self.flat(x) 
        return x

class Flatten(nn.Module):
    def forward(self, input):
        return input.view(input.size(0), -1)<jupyter_output><empty_output><jupyter_text>We create a data example to see the output shape of each block.<jupyter_code>net = Net()
X = torch.rand(size=(1,1,224,224))
for layer in net.children():
    X = layer(X)
    print(layer.__class__.__name__,'output shape:\t', X.shape)<jupyter_output>Sequential output shape:	 torch.Size([1, 96, 54, 54])
MaxPool2d output shape:	 torch.Size([1, 96, 26, 26])
Sequential output shape:	 torch.Size([1, 256, 26, 26])
MaxPool2d output shape:	 torch.Size([1, 256, 12, 12])
Sequential output shape:	 torch.Size([1, 384, 12, 12])
MaxPool2d output shape:	 torch.Size([1, 384, 5, 5])
Dropout2d output shape:	 torch.Size([1, 384, 5, 5])
Sequential output shape:	 torch.Size([1, 10, 5, 5])
AdaptiveMaxPool2d output shape:	 torch.Size([1, 10, 1, 1])
Flatten output shape:	 torch.Size([1, 10])<jupyter_text>Data Acquisition and TrainingAs before we use Fashion-MNIST to train the model.NiN's training is similar to that for AlexNet and VGG,but it often uses a larger learning rate.<jupyter_code>lr, num_epochs, batch_size, device = 0.1, 5, 128, d2l.try_gpu()

#Xavier initialization of weights
def init_weights(m):
    if type(m) == nn.Linear or type(m) == nn.Conv2d:
        torch.nn.init.xavier_uniform_(m.weight)

net.apply(init_weights)

#Loading fashion-MNIST data
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size, resize=224)

#criterion
criterion = nn.CrossEntropyLoss()

d2l.train_ch5(net, train_iter, test_iter, criterion, num_epochs, batch_size, device, lr)<jupyter_output>training on cuda:0
epoch 1, loss 0.0179, train acc 0.102, test acc 0.100, time 70.8 sec
epoch 2, loss 0.0115, train acc 0.458, test acc 0.653, time 69.7 sec
epoch 3, loss 0.0073, train acc 0.669, test acc 0.709, time 69.9 sec
epoch 4, loss 0.0062, train acc 0.708, test acc 0.716, time 70.1 sec
epoch 5, loss 0.0045, train acc 0.788, test acc 0.792, time 70.2 sec
{'fraction_non_alphanumeric': 0.09130434782608696, 'fraction_numerical': 0.042278860569715145, 'mean_word_length': 4.432410423452769, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '661161', 'license': 'Apache-2.0', 'n_tokens_mistral': 2259, 'n_tokens_neox': 2052, 'n_words': 746, 'path': 'Ch09_Modern_Convolutional_Networks/Network_in_Network(NiN).ipynb', 'repo_name': 'wangli320/d2l-pytorch', 'size': '6673'}