<jupyter_start><jupyter_text>Fully Convolutional Networks:label:`sec_fcn`As discussed in :numref:`sec_semantic_segmentation`,semantic segmentationclassifies images in pixel level.A fully convolutional network (FCN)uses a convolutional neural network totransform image pixels to pixel classes :cite:`Long.Shelhamer.Darrell.2015`.Unlike the CNNs that we encountered earlierfor image classification or object detection,a fully convolutional networktransforms the height and width of intermediate feature mapsback to those of the input image:this is achieved bythe transposed convolutional layerintroduced in :numref:`sec_transposed_conv`.As a result,the classification outputand the input image have a one-to-one correspondence in pixel level:the channel dimension at any output pixel holds the classification resultsfor the input pixel at the same spatial position.<jupyter_code>%matplotlib inline
from mxnet import gluon, image, init, np, npx
from mxnet.gluon import nn
from d2l import mxnet as d2l

npx.set_np()<jupyter_output><empty_output><jupyter_text>The ModelHere we describe the basic design of the fully convolutional network model. As shown in :numref:`fig_fcn`,this model first uses a CNN to extract image features,then transforms the number of channels intothe number of classesvia a $1\times 1$ convolutional layer,and finally transforms the height and width ofthe feature mapsto thoseof the input image viathe transposed convolution introduced in :numref:`sec_transposed_conv`. As a result,the model output has the same height and width as the input image,where the output channel contains the predicted classesfor the input pixel at the same spatial position.![Fully convolutional network.](../img/fcn.svg):label:`fig_fcn`Below, we [**use a ResNet-18 model pretrained on the ImageNet dataset to extract image features**]and denote the model instance as `pretrained_net`.The last few layers of this modelinclude a global average pooling layerand a fully-connected layer:they are not neededin the fully convolutional network.<jupyter_code>pretrained_net = gluon.model_zoo.vision.resnet18_v2(pretrained=True)
pretrained_net.features[-3:], pretrained_net.output<jupyter_output><empty_output><jupyter_text>Next, we [**create the fully convolutional network instance `net`**].It copies all the pretrained layers in the ResNet-18except for the final global average pooling layerand the fully-connected layer that are closestto the output.<jupyter_code>net = nn.HybridSequential()
for layer in pretrained_net.features[:-2]:
    net.add(layer)<jupyter_output><empty_output><jupyter_text>Given an input with height and width of 320 and 480 respectively,the forward propagation of `net`reduces the input height and width to 1/32 of the original, namely 10 and 15.<jupyter_code>X = np.random.uniform(size=(1, 3, 320, 480))
net(X).shape<jupyter_output><empty_output><jupyter_text>Next, we [**use a $1\times 1$ convolutional layer to transform the number of output channels into the number of classes (21) of the Pascal VOC2012 dataset.**]Finally, we need to (**increase the height and width of the feature maps by 32 times**) to change them back to the height and width of the input image. Recall how to calculate the output shape of a convolutional layer in :numref:`sec_padding`. Since $(320-64+16\times2+32)/32=10$ and $(480-64+16\times2+32)/32=15$, we construct a transposed convolutional layer with stride of $32$, settingthe height and width of the kernelto $64$, the padding to $16$.In general,we can see thatfor stride $s$,padding $s/2$ (assuming $s/2$ is an integer),and the height and width of the kernel $2s$, the transposed convolution will increasethe height and width of the input by $s$ times.<jupyter_code>num_classes = 21
net.add(nn.Conv2D(num_classes, kernel_size=1),
        nn.Conv2DTranspose(
            num_classes, kernel_size=64, padding=16, strides=32))<jupyter_output><empty_output><jupyter_text>[**Initializing Transposed Convolutional Layers**]We already know thattransposed convolutional layers can increasethe height and width offeature maps.In image processing, we may need to scale upan image, i.e., *upsampling*.*Bilinear interpolation*is one of the commonly used upsampling techniques.It is also often used for initializing transposed convolutional layers.To explain bilinear interpolation,say that given an input imagewe want to calculate each pixel of the upsampled output image.In order to calculate the pixel of the output imageat coordinate $(x, y)$, first map $(x, y)$ to coordinate $(x', y')$ on the input image, for example, according to the ratio of the input size to the output size. Note that the mapped $x′$ and $y′$ are real numbers. Then, find the four pixels closest to coordinate$(x', y')$ on the input image. Finally, the pixel of the output image at coordinate $(x, y)$ is calculated based on these four closest pixelson the input image and their relative distance from $(x', y')$. Upsampling of bilinear interpolationcan be implemented by the transposed convolutional layer with the kernel constructed by the following `bilinear_kernel` function. Due to space limitations, we only provide the implementation of the `bilinear_kernel` function belowwithout discussions on its algorithm design.<jupyter_code>def bilinear_kernel(in_channels, out_channels, kernel_size):
    factor = (kernel_size + 1) // 2
    if kernel_size % 2 == 1:
        center = factor - 1
    else:
        center = factor - 0.5
    og = (np.arange(kernel_size).reshape(-1, 1),
          np.arange(kernel_size).reshape(1, -1))
    filt = (1 - np.abs(og[0] - center) / factor) * \
           (1 - np.abs(og[1] - center) / factor)
    weight = np.zeros((in_channels, out_channels, kernel_size, kernel_size))
    weight[range(in_channels), range(out_channels), :, :] = filt
    return np.array(weight)<jupyter_output><empty_output><jupyter_text>Let us [**experiment with upsampling of bilinear interpolation**] that is implemented by a transposed convolutional layer. We construct a transposed convolutional layer that doubles the height and weight,and initialize its kernel with the `bilinear_kernel` function.<jupyter_code>conv_trans = nn.Conv2DTranspose(3, kernel_size=4, padding=1, strides=2)
conv_trans.initialize(init.Constant(bilinear_kernel(3, 3, 4)))<jupyter_output><empty_output><jupyter_text>Read the image `X` and assign the upsampling output to `Y`. In order to print the image, we need to adjust the position of the channel dimension.<jupyter_code>img = image.imread('../img/catdog.jpg')
X = np.expand_dims(img.astype('float32').transpose(2, 0, 1), axis=0) / 255
Y = conv_trans(X)
out_img = Y[0].transpose(1, 2, 0)<jupyter_output><empty_output><jupyter_text>As we can see, the transposed convolutional layer increases both the height and width of the image by a factor of two.Except for the different scales in coordinates,the image scaled up by bilinear interpolation and the original image printed in :numref:`sec_bbox` look the same.<jupyter_code>d2l.set_figsize()
print('input image shape:', img.shape)
d2l.plt.imshow(img.asnumpy());
print('output image shape:', out_img.shape)
d2l.plt.imshow(out_img.asnumpy());<jupyter_output>input image shape: (561, 728, 3)
output image shape: (1122, 1456, 3)<jupyter_text>In a fully convolutional network, we [**initialize the transposed convolutional layer with upsampling of bilinear interpolation. For the $1\times 1$ convolutional layer, we use Xavier initialization.**]<jupyter_code>W = bilinear_kernel(num_classes, num_classes, 64)
net[-1].initialize(init.Constant(W))
net[-2].initialize(init=init.Xavier())<jupyter_output><empty_output><jupyter_text>[**Reading the Dataset**]We readthe semantic segmentation datasetas introduced in :numref:`sec_semantic_segmentation`. The output image shape of random cropping isspecified as $320\times 480$: both the height and width are divisible by $32$.<jupyter_code>batch_size, crop_size = 32, (320, 480)
train_iter, test_iter = d2l.load_data_voc(batch_size, crop_size)<jupyter_output>read 1114 examples<jupyter_text>[**Training**]Now we can train our constructedfully convolutional network. The loss function and accuracy calculation hereare not essentially different from those in image classification of earlier chapters. Because we use the output channel of thetransposed convolutional layer topredict the class for each pixel,the channel dimension is specified in the loss calculation.In addition, the accuracy is calculatedbased on correctnessof the predicted class for all the pixels.<jupyter_code>num_epochs, lr, wd, devices = 5, 0.1, 1e-3, d2l.try_all_gpus()
loss = gluon.loss.SoftmaxCrossEntropyLoss(axis=1)
net.collect_params().reset_ctx(devices)
trainer = gluon.Trainer(net.collect_params(), 'sgd',
                        {'learning_rate': lr, 'wd': wd})
d2l.train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs, devices)<jupyter_output>loss 0.327, train acc 0.892, test acc 0.851
182.3 examples/sec on [gpu(0), gpu(1)]<jupyter_text>[**Prediction**]When predicting, we need to standardize the input imagein each channel and transform the image into the four-dimensional input format required by the CNN.<jupyter_code>def predict(img):
    X = test_iter._dataset.normalize_image(img)
    X = np.expand_dims(X.transpose(2, 0, 1), axis=0)
    pred = net(X.as_in_ctx(devices[0])).argmax(axis=1)
    return pred.reshape(pred.shape[1], pred.shape[2])<jupyter_output><empty_output><jupyter_text>To [**visualize the predicted class**] of each pixel, we map the predicted class back to its label color in the dataset.<jupyter_code>def label2image(pred):
    colormap = np.array(d2l.VOC_COLORMAP, ctx=devices[0], dtype='uint8')
    X = pred.astype('int32')
    return colormap[X, :]<jupyter_output><empty_output><jupyter_text>Images in the test dataset vary in size and shape.Since the model uses a transposed convolutional layer with stride of 32,when the height or width of an input image is indivisible by 32,the output height or width of thetransposed convolutional layer will deviate from the shape of the input image.In order to address this issue,we can crop multiple rectangular areas with height and width that are integer multiples of 32 in the image,and perform forward propagationon the pixels in these areas separately.Note thatthe union of these rectangular areas needs to completely cover the input image.When a pixel is covered by multiple rectangular areas,the average of the transposed convolution outputsin separate areas for this same pixelcan be input tothe softmax operationto predict the class.For simplicity, we only read a few larger test images,and crop a $320\times480$ area for prediction starting from the upper-left corner of an image.For these test images, weprint their cropped areas,prediction results,and ground-truth row by row.<jupyter_code>voc_dir = d2l.download_extract('voc2012', 'VOCdevkit/VOC2012')
test_images, test_labels = d2l.read_voc_images(voc_dir, False)
n, imgs = 4, []
for i in range(n):
    crop_rect = (0, 0, 480, 320)
    X = image.fixed_crop(test_images[i], *crop_rect)
    pred = label2image(predict(X))
    imgs += [X, pred, image.fixed_crop(test_labels[i], *crop_rect)]
d2l.show_images(imgs[::3] + imgs[1::3] + imgs[2::3], 3, n, scale=2);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08736571073426264, 'fraction_numerical': 0.023084435763118173, 'mean_word_length': 5.79372738238842, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '385606', 'license': 'MIT', 'n_tokens_mistral': 3500, 'n_tokens_neox': 3243, 'n_words': 1254, 'path': 'python/d2l-en/mxnet/chapter_computer-vision/fcn.ipynb', 'repo_name': 'rtp-aws/devpost_aws_disaster_recovery', 'size': '11263'}