Simple and Efficient Architectures for Semantic Segmentation

Though the state-of-the architectures for semantic segmentation, such as HRNet, demonstrate impressive accuracy, the complexity arising from their salient design choices hinders a range of model acceleration tools, and further they make use of operations that are inefficient on current hardware. This paper demonstrates that a simple encoder-decoder architecture with a ResNet-like backbone and a small multi-scale head, performs on-par or better than complex semantic segmentation architectures such as HRNet, FANet and DDRNets. Naively applying deep backbones designed for Image Classification to the task of Semantic Segmentation leads to sub-par results, owing to a much smaller effective receptive field of these backbones. Implicit among the various design choices put forth in works like HRNet, DDRNet, and FANet are networks with a large effective receptive field. It is natural to ask if a simple encoder-decoder architecture would compare favorably if comprised of backbones that have a larger effective receptive field, though without the use of inefficient operations like dilated convolutions. We show that with minor and inexpensive modifications to ResNets, enlarging the receptive field, very simple and competitive baselines can be created for Semantic Segmentation. We present a family of such simple architectures for desktop as well as mobile targets, which match or exceed the performance of complex models on the Cityscapes dataset. We hope that our work provides simple yet effective baselines for practitioners to develop efficient semantic segmentation models.


Introduction
Convolutional Neural Network based approaches to Semantic Image Segmentation have seen a lot of progress in the past years in terms of architectures, training techniques, Figure 1. FFNet Architecture comprises of a backbone (encoder), in this case ResNet-like, feeding into a compact multi-branch Uphead (decoder), that subsequently feeds multi-scale features to the task specific head. 's' indicates the block or layer stride. The choices for the stem, widths & depths of the backbone blocks, widths of the convolutions in the Up-head, choice of upsampling operator (bilinear vs nearest), and the design of the task head depend on the target platform and the task. The backbone options considered in this paper are listed in Table 1, and the stem, Uphead and Segmentation-head options are depicted in Figure 2. datasets and loss functions. The extent of improvement arising from various aspects of the improved CNN architectures can often get muddled due to the specifics of training, datasets and training pipeline differences.
For semantic image segmentation of natural images, researchers and practitioners who are new to the field look to state-of-the-art tables [2] and available code bases [1] as a starting point, and build on top [27] of these. Unfortunately, these benchmarks happen to be dominated by increasingly complex and costly architectures [10,22,26].
We set out to examine the extent of efficacy of these complex designs over a conceptually simpler architecture. The baseline architecture we use is most closely related to FPN [15], originally proposed for object detection. Specifically, we investigate whether the accuracy gains of the more complex architectures hold up when their associated non-architectural improvements are applied to this simple baseline architecture. We see that when using ResNet-50/101 [9] backbones comprised of bottleneck-blocks in an FPN-like design the network is indeed markedly worse than the more complex designs. In this paper, we conjec- Figure 2. Various choices for the Stem, Up-head, and Segmentation-head that we consider in this paper. These are connected to backbone networks of varying widths and depths, listed in Table 1. For GPU models, bilinear upsampling is used. For mobile models, nearest neighbour upsampling is used. The choices depicted are not comprehensive, and only meant to represent a few distinct points in a potential NAS search space. In the text we will refer to the Stem, Up-head and Segmentation-head choice combinations as A-B-B, C-B-C etc.
ture that this drop in performance is mainly due to the decrease in receptive field due to using bottleneck-blocks [5]. We show that with similarly deep ResNet backbones comprised of basic-blocks, with a consequently much larger receptive field, the simple architecture does compare favorably to the more complex designs. Our simple architecture named FFNet not only reduces the inference time and compute costs, but is comprised entirely of operations well supported on a wide variety of hardware, further making on-device deployment easier.
We describe the FFNet meta-architecture in Section 2, and present results for a family of FFNet models in Section 3. Given the extent of the experiments and the compute requirements, we limit the experiments to the Cityscapes dataset.

FFNet: A Simple Architecture
In Figure 1, we depict a schema of our Fuss-Free Network (FFNet); a simple architecture inspired by the FPN architecture. It has an encoder-decoder structure. The encoder is comprised of a ResNet backbone without the classification head. Instead of just using features from the very last layer of the backbone, features at different spatial resolutions are extracted from all intermediate residual stages. These features are passed on to a light convolutional decoder head that upsamples and incorporates features from lower resolution branches to higher resolution branches. This decoder head, which we will refer to as 'Uphead', also outputs features at different spatial resolutions. These multi-scale features are subsequently used as input to a small task-specific head, such as for segmentation or classification.
A lot of flexibility exists within the general setup of FFNet. We can freely change the width, depth, and the type of backbone building blocks, number of feature scales, the head type and the head width. Figure 2 depicts the various stem, Up-head, and Segmentation-head choices we investigate in this paper, marked as A/B/C. These are combined with various backbone width and depth configurations depicted in Table 1, depending on the target platform. The first residual block in the backbone can have a stride of 1 or 2, which changes the spatial resolution of the output. We show results for both in Section 3. Specific instantiations of FFNets throughout the paper will refer to a backbone, such as ResNet-50, paired with a Stem-Up-Seg combination denoted as A-B-B.
We show models for desktop GPU and mobile targets. The desktop models use bilinear upsampling in the Uphead, while mobile models use nearest neighbour upsampling.
This is a simple design with no particular restrictions imposed on the number of stages in the backbone or the number of feature scales being output, the choice of striding in the stem and the backbone. The architecture can thus easily be adapted for other tasks.

Experimental Setup
Dataset. The Cityscapes [7] dataset is comprised of videos of urban street scenes recorded from a moving vehicle. 5k image frames have high quality pixel-level annotations of 19 semantic classes, and a further 20k frames that are weakly annotated.
Training Setup. Since the number of annotated frames in the Cityscapes dataset is limited, we utilize ImageNet pretraining of the models. The FFNet models are pretrained on the ILSVRC2012 [8] dataset in the FFNet form, using the classification head from HRNet [22] in place of the segmentation head. For ImageNet pretraining, we make use of Pytorch Image Models [23], and train for 150 epochs.
For Cityscapes training, the stem and the backbone are initialized from the ImageNet pretraining. The Up-head and the Segmentation-head are initialized from scratch. Since the fine grained annotations on the Cityscapes dataset are Table 1. Configurations of backbones examined in the paper. The naming convention follows that of ResNets [9], even though these are paired up with different stems and heads depicted in Figure 2, which changes the overall number of layers. limited to 5k images, learning meaningful and generalizable feature representations with deep CNNs can be challenging. We use pseudo labels from Tao et al. [21] on the images with coarse annotations. Additionally, we use the RMI [28] loss as an auxiliary loss while training. The models are trained for 175 epochs with a batch size of 8. We will make the ImageNet and Cityscapes training hyperparameters available with the models.

Experiments and Comparisons
We examine various configurations of FFNets. The backbones of different depths, widths, and block types that we tried are detailed in Table 1. These backbones are paired with various stem, Up-head, and Segmentation options from Figure 2, and the configurations referred to as A-B-B etc.
For desktop GPU FFNets, we use input images of size 1024 × 2048. We show FFNet models with a stride of 1 and 2 in the first residual stage of the ResNet backbone, with an output segmentation map resolution of 256 × 512 and 128 × 256 respectively. We'll refer to these models as FFNet-GPU-Large and FFNet-GPU-Small respectively.
For mobile FFNets, we use a stride of 2 in the first residual stage, with an input resolution of 1024 × 2048, and a stride of 1 when the input resolution is 512 × 1024, both resulting in a segmentation map resolution of 128 × 256.

Efficacy of Basic Blocks vs Bottleneck Blocks
FFNets with backbones comprised of basic-blocks far outperform backbones comprised of bottleneck blocks, for FFNet-GPU-Large and FFNet-GPU-Small models. See FFNet A-A-A in Figure 3 and Figure 5. Prior work has primarily used ResNet-101 and ResNet-50 as backbones in baselines for Segmentation, and seen poor results on account of an inadequately large receptive field, and needing to enlarge the receptive field through tricks like dilated convolution [5]. New backbones like ResNet-150, 134, 86, 56 described in Table 1, and known backbones like ResNet-34, which are all comprised of basic-blocks fare far better than ResNet-50 and ResNet-101, particularly in the case of FFNet-GPU-Large which operates on larger feature maps.

Exploring the Model Space
While the effects of varying the backbone depth are obvious and expected, we also experiment with varying the width of the backbone, and the width of the Up-head and the Segmentation-head. For FFNet-GPU-Large models, given the larger spatial resolution of the output feature maps, significant speed improvements arise from narrowing the Uphead and the Segmentation-head. FFNet A-B-B (cyan) in Figure 3, which uses narrower Up-head and Segmentationhead, provides a better speed-accuracy trade-off than FFNet A-A-A (blue) with its wider heads. For FFNet-GPU-Small  Table 2.

Models for Mobile
For FFNet models intended for mobile deployment, we utilize narrower ResNet backbones and heads. Instead of bilinear upsampling in the heads, FFNet-Mobile models use nearest neighbour upsampling. We show a family of models with their inference times on a Samsung S21 DSP, in Figure 6 and Table 4. Changing the input resolution, and the width of the head allows on-device models to be created with a better speed-accuracy tradeoff. See black, blue and cyan plots in Figure 6.

3-Stage Backbone
We experiment with the backbone structure, beyond the depth and width of the backbone, in order to create a more efficient FFNet. Typical ResNets are comprised of 4 residual stages. The first stage operates on the largest feature maps, and has a significant compute and memory cost. We create a 3-stage ResNet backbone, removing the first residual stage. For FFNets, the 3-stage backbone interfaces with the Up-head, using the output from the stem instead of the output from the first residual stage of a 4-stage backbone. See  NS C-C-C (light green) in Figure 6.
It is unclear if the 3 stage model would also be better at higher accuracies, but this experimentation reveals a new dimension of exploration to be covered in a potential NAS search space.

Comparisons to Prior Art
We use HRNet [22], DDRNet [10], and FANet [12] as our points of comparison to prior work. HRNets are commonplace in much of semantic segmentation work. DDR-Net demonstrates better performance than several classes of GPU efficient networks for Semantic Segmentation. FANets also claim GPU efficiency, and are the closest to our design, but make use of self-attention in the head. We compare the accuracy and GPU inference timings for the models, and refrain from making FLOP comparisons for a few reasons. Various libraries for FLOP computation are not able to correctly handle all kinds of operations, and may need operations implemented a certain way to correctly account for the FLOPs. Further, several operations such as tensor concatenation and channel shuffling do not have a compute cost associated, but incur a large memory access time cost. Hence, we restrict ourselves to inference time comparisons. We retrain all the models that we compare to, using the same training scheme, to make comparisons fair. Hence, the performance reported for these models is much higher than that in their respective papers. However, there may yet be better accuracy possible with model specific training-hyperparameter tuning. The validation accuracy is given for single scale inference, with no test-time augmentation used.
Comparison with HRNet: HRNet utilizes multi-scale feature representations throughout the network, with repeated information exchange across the feature scales, and a multi-scale feature output that is used by the task specific head. FFNet A-A-A (blue) models using backbones comprised of basic-blocks perform on par with HRNets (red), as shown in Figure 3, while FFNet A-A-A (violet) models using bottleneck-blocks are markedly worse. A key feature of HRNet that is not remarked upon, is that the network is 75+ layers deep, and comprised of basic-blocks. It consequently has a larger effective receptive field than bottleneck-block based networks like ResNet-50/101. We find that a multi-scale feature output constructed with the simple FFNet architectures achieves similar accuracy, when the backbones are made of basic-blocks. These results point to the fact that the large receptive field is a critical contributor to the overall accuracy, and comparable results are possible with simple designs like the FFNets, without requiring other components such as a repeated information exchange across scales. It would need to be determined when and if the repeated information exchange across feature scales is effective. Other families of FFNets with narrower heads and backbones can perform better than HRNets as seen in Table 2.
tectures such as DDRNet [10] and BiseNet [24,25] have followed HRNet, claiming that it is important to maintain higher resolution information throughout the network, as well inter-branch information exchange. They propose to make such a design faster through reducing the width of the backbone, and maintaining only one higher resolution feature scale instead of multiple. They too, however, use a backbone with a large receptive field. We show comparisons with DDRNet, which has a better reported performance than BiseNets.
FFNet A-A-A (blue) models using backbones comprised of basic-blocks also perform on par with DDRNets (red), as shown in Figure 5. As with the case of HRNets, this demonstrates that with a suitably large receptive field, simple encoder-decoder models can work as well as 2-stream models like DDRNets. By changing the backbone and head width of FFNets (blue, black), models that are better than DDRNets can be found for all speed-accuracy tradeoffs. See Table 3. The 3-stage FFNet-GPU-Large models (green) also outperform DDRNets.
Comparison with FANet: Approaches like FANet [12] and CCNet [13] that make use of attention, could be viewed as attempts to increase the effective receptive field of the network, when using backbones with small effective receptive fields. Indeed, FANets with bottleneck-blocks in the backbone are comparable to FANets with basic-blocks in the backbone. See Figure 5.
One of the questions that it naturally leads to is whether an equivalent compute cost increase through enlarging the receptive field of backbone delivers the same accuracy boost that incorporating these attention-based structures does. As shown in Figure 5 and Table 3, FFNet A-A-A (blue) models with a backbone with an adequately large receptive are on-par with FANet (orange, brown) models, and the selfattention head of FANet provides diminishing returns.

Related Work
For a comprehensive survey of the developments in Deep Learning based Semantic Segmentation, see Minaee Figure 6. Inference latency vs Cityscapes validation accuracy for FFNet-Mobile Models on Samsung S21: The models run in realtime on the mobile DSP. Regardless of input resolution, all models output segmentation maps at the same resolution. Using a smaller input resolution (blue, cyan) affords a better speed-accuracy tradeoff than a higher input resolution with additional downsampling in the backbone (black). Narrowing the head B-B-B (blue) → B-C-C (cyan) offers a better speed-accuracy tradeoff. As for desktop GPUs, 3-stage backbones (green, light green) are more efficient than 4-stage models for mobile too. See Table 4. et al. [17]. Here we only cover aspects directly relevant to our discussion. The go-to architectures for image segmentation roughly belong to two categories. Encoder-decoder designs like UNet [20], DeepLabv1/v2/v3 [5,6], SegNet [3], with a downsampling backbone (encoder) and an upsampling head (decoder), and HRNet-like [10,22,26], designs which maintain features at multiple resolutions throughout the network. HRNet-like multi resolution networks typically dominate the results tables, with claims of higher accuracy.
Encoder-decoder designs can have similar sized backbone and head, such as for UNet, or make use of bigger ResNet-like backbones, with various smaller head designs that enable context aggregation, such as for DeepLabv3. DeepLabv1/v2/v3 utilize dilated convolutions in the ResNet-like backbones to increase the receptive field size. Dilated convolutions are not efficient on existing hardware. Recent designs such as MobileNetv3 [11] reduce the use of dilated convolutions by restricting them only to the last block, but do not entirely eliminate them. Kirillov et al. [14] show that the FPN [15] design with an un-modified ResNet [9] backbone is competitive with DeepLab [5] like designs which remove striding and employ dilated convolutions in the ResNet backbone. Panoptic-FPN [14] however still employs the ResNet backbones using bottleneck-  [6], with the goal to capture features of different scales. ASPP achieves this through convolutions with different dilations. We use a very simple decoder head, and future investigations can determine if substituting it with other head designs is helpful. As discussed in Section 3.5, the simple FFNet architecture performs on par with HRNet-like architectures like HRNet [22], DDRNet [10], and GPU-efficient encoderdecoder architectures like FANet [12].

Discussion
We have only taken a limited look at the avenues of efficiency improvements that can be explored to make FFNets even faster. Various approaches to efficient and real-time image segmentation [10,16,[24][25][26] make use of FLOP ef-ficient operations and narrower layers, but still within the context of complex meta-architectures. See the related work sections of Hong et al. [10] and Yu et al. [26] for a detailed discussion of efficient segmentation architectures. These approaches could also be applied to FFNets.
Ideally, the space of efficient architectural configurations should be explored using an automated approach, such as DONNA [19] or LANA [18]. Here too, the simplicity of the design of the meta-architecture is a significant advantage, because the architecture can be trivially split into a small number of simple blocks that can be handled by NAS approaches that rely on blockwise knowledgedistillation [4,18,19]. HRNet, DDRNet etc. either split into a small number of complex blocks, or a very large number of simple blocks, and handling these with blockwise NAS approaches becomes untenable.
Although our study only looks at road-scene semantic segmentation, the simple FFNet architectures could very well be competitive baselines for other computer vision tasks, as well as other semantic segmentation datasets.

Conclusion
We show that simple FPN-inspired baselines for Semantic Image Segmentation are efficient and extremely competitive with SoTA architectures across a variety of devices, ranging from highly accurate desktop GPU models to highly efficient mobile models. We also show that specific architecture instances designed for ImageNet are not necessarily the best for other tasks, and there exist better task specific architectures within the same design space. It is helpful to think about the specific requirements of the task before deciding to port a network from another task. We hope that this manuscript convinces the reader about the unexplored potential of simpler CNN architectures for semantic segmentation, and beyond.