<jupyter_start><jupyter_text>6.3. Padding and StrideSource : https://d2l.ai/chapter_convolutional-neural-networks/padding-and-strides.html6.3.1. PaddingIn the following example, we create a two-dimensional convolutional layer with a height and width of 3 and apply 1 pixel of padding on all sides. Given an input with a height and width of 8, we find that the height and width of the output is also 8.<jupyter_code>!pip install d2l
import torch
from torch import nn


# We define a convenience function to calculate the convolutional layer. This
# function initializes the convolutional layer weights and performs
# corresponding dimensionality elevations and reductions on the input and
# output
def comp_conv2d(conv2d, X):
    # Here (1, 1) indicates that the batch size and the number of channels
    # are both 1
    X = X.reshape((1, 1) + X.shape)
    Y = conv2d(X)
    # Exclude the first two dimensions that do not interest us: examples and
    # channels
    return Y.reshape(Y.shape[2:])
# Note that here 1 row or column is padded on either side, so a total of 2
# rows or columns are added
conv2d = nn.Conv2d(1, 1, kernel_size=3, padding=1)
X = torch.rand(size=(8, 8))
comp_conv2d(conv2d, X).shape<jupyter_output><empty_output><jupyter_text>When the height and width of the convolution kernel are different, we can make the output and input have the same height and width by setting different padding numbers for height and width.<jupyter_code># Here, we use a convolution kernel with a height of 5 and a width of 3. The
# padding numbers on either side of the height and width are 2 and 1,
# respectively
conv2d = nn.Conv2d(1, 1, kernel_size=(5, 3), padding=(2, 1))
comp_conv2d(conv2d, X).shape<jupyter_output><empty_output><jupyter_text>6.3.2. StrideBelow, we set the strides on both the height and width to 2, thus halving the input height and width.<jupyter_code>conv2d = nn.Conv2d(1, 1, kernel_size=3, padding=1, stride=2)
comp_conv2d(conv2d, X).shape
#Next, we will look at a slightly more complicated example.
conv2d = nn.Conv2d(1, 1, kernel_size=(3, 5), padding=(0, 1), stride=(3, 4))
comp_conv2d(conv2d, X).shape<jupyter_output><empty_output><jupyter_text>**6.3.3. Summary**1. Padding can increase the height and width of the output. This is often used to give the output the same height and width as the input.2. The stride can reduce the resolution of the output, for example reducing the height and width of the output to only  1/n  of the height and width of the input ( n  is an integer greater than  1 ).3. Padding and stride can be used to adjust the dimensionality of the data effectively.  6.4. Multiple Input and Multiple Output Channels**6.4.1. Multiple Input Channel**sTo make sure we really understand what is going on here, we can implement cross-correlation operations with multiple input channels ourselves. Notice that all we are doing is performing one cross-correlation operation per channel and then adding up the results.<jupyter_code>import torch
from d2l import torch as d2l

def corr2d_multi_in(X, K):
    # First, iterate through the 0th dimension (channel dimension) of `X` and
    # `K`. Then, add them together
    return sum(d2l.corr2d(x, k) for x, k in zip(X, K))<jupyter_output><empty_output><jupyter_text>We can construct the input tensor X and the kernel tensor K corresponding to the values in Fig. 6.4.1 to validate the output of the cross-correlation operation.<jupyter_code>X = torch.tensor([[[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]],
               [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]])
K = torch.tensor([[[0.0, 1.0], [2.0, 3.0]], [[1.0, 2.0], [3.0, 4.0]]])

corr2d_multi_in(X, K)<jupyter_output><empty_output><jupyter_text>**6.4.2. Multiple Output Channels**We implement a cross-correlation function to calculate the output of multiple channels as shown below.<jupyter_code>def corr2d_multi_in_out(X, K):
    # Iterate through the 0th dimension of `K`, and each time, perform
    # cross-correlation operations with input `X`. All of the results are
    # stacked together
    return torch.stack([corr2d_multi_in(X, k) for k in K], 0)<jupyter_output><empty_output><jupyter_text>We construct a convolution kernel with 3 output channels by concatenating the kernel tensor K with K+1 (plus one for each element in K) and K+2.<jupyter_code>K = torch.stack((K, K + 1, K + 2), 0)
K.shape<jupyter_output><empty_output><jupyter_text>Below, we perform cross-correlation operations on the input tensor X with the kernel tensor K. Now the output contains 3 channels. The result of the first channel is consistent with the result of the previous input tensor X and the multi-input channel, single-output channel kernel<jupyter_code>corr2d_multi_in_out(X, K)<jupyter_output><empty_output><jupyter_text>**6.4.3.  1×1  Convolutional Layer**Let us check whether this works in practice: we implement a  1×1  convolution using a fully-connected layer. The only thing is that we need to make some adjustments to the data shape before and after the matrix multiplication.<jupyter_code>def corr2d_multi_in_out_1x1(X, K):
    c_i, h, w = X.shape
    c_o = K.shape[0]
    X = X.reshape((c_i, h * w))
    K = K.reshape((c_o, c_i))
    # Matrix multiplication in the fully-connected layer
    Y = torch.matmul(K, X)
    return Y.reshape((c_o, h, w))<jupyter_output><empty_output><jupyter_text>When performing  1×1  convolution, the above function is equivalent to the previously implemented cross-correlation function corr2d_multi_in_out. Let us check this with some sample data.<jupyter_code>X = torch.normal(0, 1, (3, 3, 3))
K = torch.normal(0, 1, (2, 3, 1, 1))

Y1 = corr2d_multi_in_out_1x1(X, K)
Y2 = corr2d_multi_in_out(X, K)
assert float(torch.abs(Y1 - Y2).sum()) < 1e-6<jupyter_output><empty_output><jupyter_text>**6.4.4. Summary**1. Multiple channels can be used to extend the model parameters of the convolutional layer.2. The  1×1  convolutional layer is equivalent to the fully-connected layer, when applied on a per pixel basis.3. The  1×1  convolutional layer is typically used to adjust the number of channels between network layers and to control model complexity. 6.5. Pooling**6.5.1. Maximum Pooling and Average Pooling**In the code below, we implement the forward propagation of the pooling layer in the pool2d function. This function is similar to the corr2d function in Section 6.2. However, here we have no kernel, computing the output as either the maximum or the average of each region in the input.<jupyter_code>import torch
from torch import nn
from d2l import torch as d2l

def pool2d(X, pool_size, mode='max'):
    p_h, p_w = pool_size
    Y = torch.zeros((X.shape[0] - p_h + 1, X.shape[1] - p_w + 1))
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
            if mode == 'max':
                Y[i, j] = X[i: i + p_h, j: j + p_w].max()
            elif mode == 'avg':
                Y[i, j] = X[i: i + p_h, j: j + p_w].mean()
    return Y<jupyter_output><empty_output><jupyter_text>We can construct the input tensor X in Fig. 6.5.1 to validate the output of the two-dimensional maximum pooling layer.<jupyter_code>X = torch.tensor([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]])
pool2d(X, (2, 2))<jupyter_output><empty_output><jupyter_text>Also, we experiment with the average pooling layer.<jupyter_code>pool2d(X, (2, 2), 'avg')<jupyter_output><empty_output><jupyter_text>6.5.2. Padding and StrideAs with convolutional layers, pooling layers can also change the output shape.And as before, we can alter the operation to achieve a desired output shape by padding the input and adjusting the stride. We can demonstrate the use of padding and strides in pooling layers via the built-in two-dimensional maximum pooling layer from the deep learning framework. We first construct an input tensor X whose shape has four dimensions, where the number of examples (batch size) and number of channels are both 1.<jupyter_code>X = torch.arange(16, dtype=torch.float32).reshape((1, 1, 4, 4))
X<jupyter_output><empty_output><jupyter_text>By default, the stride and the pooling window in the instance from the framework’s built-in class have the same shape. Below, we use a pooling window of shape (3, 3), so we get a stride shape of (3, 3) by default.<jupyter_code>pool2d = nn.MaxPool2d(3)
pool2d(X)
#The stride and padding can be manually specified.

pool2d = nn.MaxPool2d(3, padding=1, stride=2)
pool2d(X)
#Of course, we can specify an arbitrary rectangular pooling window and specify the padding and stride for height and width, respectively.

pool2d = nn.MaxPool2d((2, 3), stride=(2, 3), padding=(0, 1))
pool2d(X)<jupyter_output><empty_output><jupyter_text>**6.5.3. Multiple Channels**When processing multi-channel input data, the pooling layer pools each input channel separately, rather than summing the inputs up over channels as in a convolutional layer. This means that the number of output channels for the pooling layer is the same as the number of input channels. Below, we will concatenate tensors X and X + 1 on the channel dimension to construct an input with 2 channels.<jupyter_code>X = torch.cat((X, X + 1), 1)
X<jupyter_output><empty_output><jupyter_text>As we can see, the number of output channels is still 2 after pooling.<jupyter_code>pool2d = nn.MaxPool2d(3, padding=1, stride=2)
pool2d(X)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09524828767123288, 'fraction_numerical': 0.03210616438356165, 'mean_word_length': 4.529585798816568, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 69, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '604883', 'license': 'Unlicense', 'n_tokens_mistral': 3147, 'n_tokens_neox': 2991, 'n_words': 1214, 'path': 'Week 11/Padding and Stride.ipynb', 'repo_name': 'Hanif-2610/Machine-Learning-Homework-Project', 'size': '9344'}