Lightweight and Progressively-Scalable Networks for Semantic Segmentation

Multi-scale learning frameworks have been regarded as a capable class of models to boost semantic segmentation. The problem nevertheless is not trivial especially for the real-world deployments, which often demand high efficiency in inference latency. In this paper, we thoroughly analyze the design of convolutional blocks (the type of convolutions and the number of channels in convolutions), and the ways of interactions across multiple scales, all from lightweight standpoint for semantic segmentation. With such in-depth comparisons, we conclude three principles, and accordingly devise Lightweight and Progressively-Scalable Networks (LPS-Net) that novelly expands the network complexity in a greedy manner. Technically, LPS-Net first capitalizes on the principles to build a tiny network. Then, LPS-Net progressively scales the tiny network to larger ones by expanding a single dimension (the number of convolutional blocks, the number of channels, or the input resolution) at one time to meet the best speed/accuracy tradeoff. Extensive experiments conducted on three datasets consistently demonstrate the superiority of LPS-Net over several efficient semantic segmentation methods. More remarkably, our LPS-Net achieves 73.4% mIoU on Cityscapes test set, with the speed of 413.5FPS on an NVIDIA GTX 1080Ti, leading to a performance improvement by 1.5% and a 65% speed-up against the state-of-the-art STDC. Code is available at \url{https://github.com/YihengZhang-CV/LPS-Net}.


Introduction
Semantic segmentation is to assign semantic labels to every pixel of an image or a video frame. With the development of deep neural networks, the stateof-the-art networks have successfully pushed the limits of semantic segmentation with remarkable performance improvements. For example, DeepLabV3+  and Hierarchical Multi-Scale Attention (Tao et al., 2020) achieve 82.1% and 85.4% mIoU on Cityscapes test set, which are almost saturated on that dataset. The recipe behind these successes originates from multi-scale learning. In the literature, the recent advances involve utilization of multi-scale learning for semantic segmentation along three different dimensions: U-shape (Chen et al., 2020;Peng et al., 2017), pyramid pooling Zhao et al., 2017), and multi-path framework (Chen et al., 2016;Tao et al., 2020). The U-shape structure hierarchically fuses the features to gradually increase the spatial resolution and naturally produce multi-scale features. The pyramid pooling methods delve into multi-scale information through executing spatial or atrous spatial pyramid pooling at multiple scales. Unlike the former two research schemes, the multi-path frameworks resize the input image to multiple resolutions or scales and feed each scale into an individual path of a deep  network. By doing so, the multi-path design places the input resolutions from high to low in parallel and explicitly maintains the high-resolution information rather than recovering from low-scale feature maps. As a result, the learnt features are potentially more capable of classifying and localizing each pixel.
We employ this elegant recipe of multi-path framework and further evolve such type of architectures with good accuracy/speed tradeoff for semantic segmentation. Our philosophies are from two perspectives: (1) lightweighting computational units for semantic segmentation, and (2) progressively scaling up the network while balancing accuracy and inference latency. We propose to explore the first problem by probing the basic unit of convolutional blocks, including the type of convolutions and the number of channels in convolutions, on the basis of several uniqueness (e.g., large feature maps, thin channel widths) in efficient semantic segmentation. Moreover, we further elaborate different ways of interaction across multiple paths with respect to the accuracy/speed tradeoff. Based on these lightweight practice, we build a tiny model, and then progressively expand the tiny model along multiple possible dimensions and select a single dimension that achieves the best tradeoff in each step to alleviate the second issue of accuracy/speed balance.
To materialize the idea, we present Lightweight and Progressively-Scalable Networks (LPS-Net) for efficient semantic segmentation. Specifically, LPS-Net bases the multi-path design upon the low-latency regime. Each path takes the resized image as the input to an individual network, which consists of stacked convolutional blocks. The networks across paths share the same structure but have independent learnable weights. The outputs from all the paths are aggregated to produce the score maps, which are upsampled via bilinear upsampling for pixel-level predictions. In an effort to achieve a lightweight and efficient architecture, we look into the basic unit of convolutional blocks and empirically suggest to purely use 3×3 convolutions with 2 n -divisible channel widths. Furthermore, we capitalize on a simple yet effective way of bilinear interpolation to encourage mutual exchange and interactions between paths. With these practical guidelines, LPS-Net first builds a tiny network and then scales the tiny network to a family of larger ones in a progressive manner. Technically, LPS-Net executes the growth of network complexity through expanding a single dimension of the number of convolutional blocks, the number of channels, or the input resolution at one time, as depicted in Figure 1. In our case, LPS-Net ensures a good balance between accuracy and inference speed during expansion, and shows the superiority over other manually-/auto-deigned models, as shown in Figure 2.
In summary, we have made the following contributions: (1) The lightweight design of convolutional blocks and the way of path interactions in multipath framework are shown capable of regarding as the practical principles for efficient semantic segmentation; (2) The exquisitely devised LPS-Net is shown able to progressively expand the network complexity while striking the right accuracy-efficiency tradeoff; (3) LPS-Net has been properly verified through extensive experiments over three datasets, and superior capability is observed on both NVIDIA GPUs and embedded devices in our experiments.

Related Works
Semantic Segmentation. With the success of convolutional neural networks (CNNs), FCN (Long et al., 2015) which deploys CNN in a fully convolutional manner enables dense semantic prediction and endto-end training for semantic segmentation. Following FCN, researchers propose various techniques, which achieve remarkable performances. For example, in order to extract multi-scale context information which is known to be critical for pixel labeling tasks (Chen et al., 2016;Ladický et al., 2009), PSPNet (Zhao et al., 2017) applies spatial pyramid pooling and DeepLab (Chen et al., 2018a,b) utilizes parallel dilated convolutions with different rates. Yuan et al. (2021) propose to use object context represented as a dense relation matrix to emphasize the contribution of object information. Multi-layer feature aggregation Fu et al., 2019;Ghiasi and Fowlkes, 2016;Lin et al., 2019;Nirkin et al., 2021;Peng et al., 2017) is performed to recover the spatial information loss caused by spatial reduction in the network. Chen et al. (2016) and Tao et al. (2020) capitalize on the multi-path framework to improve the predictions. Such efforts are made to achieve high-quality segmentation but without taking inference latency into account.
Lightweight Networks. The real-world deployments often demand accurate and efficient networks. To accelerate the inference, there have been several techniques, e.g., pruning (Han et al., 2016), factorization (Szegedy et al., 2016), depthwise separable convolution (Chollet, 2017;Howard et al., 2017) and group convolution (Krizhevsky et al., 2012), being proposed in the literature. To facilitate model deployments and speed up the inference for semantic segmentation, recent works present to manually (Fan et al., 2021;Li et al., 2019aLi et al., , 2020bYu et al., 2021aYu et al., , 2018aZhao et al., 2018) or automatically (Chen et al., 2020;Li et al., 2020aLi et al., , 2019bLin et al., 2020;Zhang et al., 2019) design lightweight networks. ICNet  employs a cascade network structure to achieve realtime segmentation. BiSeNet (Yu et al., 2021a(Yu et al., , 2018a treats the spatial details and category semantics of the images separately to obtain a lightweight network. DFANet  designs a feature reuse method to incorporate multi-level context into encoded features. SFNet (Li et al., 2020b) boosts the segmentation by broadcasting high-level features to highresolution features with cheap operations. Fan et al. (2021) propose a short-term dense concatenate module to enrich features with scalable receptive field and multi-scale information in a lightweight network. Lite-HRNet (Yu et al., 2021b) utilizes the high-resolution design pattern of HRNet  and introduces conditional channel weighting to replace costly pointwise (1 × 1) convolutions in shuffle blocks . BiAlignNet  augments the BiSeNet by employing a gated flow alignment module to align features in a bidirectional way. To automate lightweight network design, CAS (Zhang et al., 2019) presents resource constraints to achieve an accuracy/computation tradeoff and GAS (Lin et al., 2020) further integrates a graph convolution network as a communication mechanism between different blocks. Li et al. (2019b) prunes the search space with a partial order assumption to search a balanced network. Chen et al. (2020) forms a search space integrating multiresolution branches and calibrates the balance between accuracy and latency by an additional regularization. AutoRTNet  jointly optimizes network depth, downsampling strategies, and the way of feature aggregation to obtain real-time segmentation networks.
Despite having such lightweight networks, the way to balance speed and accuracy for efficient semantic segmentation is not fully explored. Unlike the works like Yu et al., 2021b) which model different resolutions with standard Residual Conv blocks, our work aims at probing the basic design of Conv blocks, including the type of convolutions and the number of channels, particularly for efficient semantic segmentation. Instead of pre-fixing two-path design in networks like BiSeNet (Yu et al., 2021a(Yu et al., , 2018a and BiAlignNet , we exquisitely devise a progressive expansion paradigm to scale up the network. Such expansion dynamically balances the depth, width, resolution, and the number of paths of the networks to strike the right accuracy-efficiency tradeoff.
Summary. Our work focuses on developing lightweight and scalable networks for efficient semantic segmentation upon the low-latency regime. The proposal of LPS-Net contributes by studying not only the practical design of convolutional blocks and the way of path interactions in multi-path framework for semantic segmentation, but also how a scalable multipath network can be nicely expanded to meet the right accuracy/speed balance.

LPS-Net
We proceed to present our core proposal, i.e., the Lightweight and Progressively-Scalable Networks (LPS-Net). Specifically, we first introduce the macro architecture in LPS-Net that employs the multi-path recipe. Then, three design principles are presented to upgrade this architecture from lightweight standpoint, including the type of convolutions, the number of channels in  Fig. 3: An overview of the design space in our proposed LPS-Net. The red block and blue block represent the convolutional block with stride=2 and stride=1, respectively. The size of output feature map is given for each stage. Please note that the 3-path network here is an example to depict the macro architecture, and the number of paths in LPS-Net is dynamically determined by the parameter R = {r i | N i=1 } which is optimized for the right tradeoff along the network expansion.
convolution, and the way of interaction across multiple paths. Based on these practical guidelines, a family of scalable LPS-Net is devised in a progressive manner by expanding a single dimension at one time to seek the best speed/accuracy tradeoff.
3.1 Macro Architecture Figure 3 depicts an overview of the macro architecture in our LPS-Net. The macro architecture is basically constructed in the multi-path design, which resizes the input image to multiple scales and feeds each scale into an individual path. Formally, given the input image with resolution H × W , LPS-Net with N paths (e.g., N = 3 in Figure 3) resizes the image to r i H × r i W , which acts as the input of the i-th path. Here r i denotes the scaling ratio. Each path is implemented as a stack of convolutional blocks in five stages, and the j-th stage contains B j blocks. The number of channels (i.e., channel width) are maintained within each stage. Similar to , the first convolutional block in Stage 1∼4 reduces the spatial dimension by a factor of two. By doing so, the resolution of output feature map from the last stage is riH 16 × riW 16 . Moreover, we place interaction modules at the end of Stage 3∼5, aiming to encourage the mutual exchange and interactions between paths. The outputs of all paths are aggregated and fed into a segmentation head to produce the score maps with num classes channels. Finally, we perform bilinear upsampling over the score maps, yielding the outputs with resolution H × W that exactly matches the input resolution. Please note that the scaling ratios of paths R = {r i | N i=1 }, the repeated number of convolutional blocks B = {B j | 5 j=1 }, and the number of channels C = {C j | 5 j=1 } in this macro architecture can be flexibly set to make the network structure tailored to the target inference time and adjust the scalability of LPS-Net.

Convolutional Block
Convolutional block is the basic computing unit in Convolutional Neural Networks. The stack of convolutional blocks naturally consumes a high percentage of inference time throughout the entire network. Therefore, lightweighting convolutional block is essential for an efficient network. In this section, we delve into the design of basic computational block in LPS-Net along two dimensions: 1) the type of convolutional block, and 2) the number of channels.
The Type of Convolutional Block. In pursuit of less computational complexity, a series of innovations have been proposed to remould convolutional blocks (Han et al., 2020;He et al., 2016;Howard et al., 2017;Zhang et al., 2018). In these works, the amount of floating-point operations (FLOPs) is often utilized as the measurement of computational complexity, which guides the design of lightweight network for image recognition. However, FLOPs ignores the memory access cost (MAC) and the degree of parallelism that  substantially impact the inference latency of networks, thereby resulting in the discrepancy between FLOPs and actual latency. More importantly, the extension of lightweight designs from image recognition to semantic segmentation is not trivial due to several uniqueness in efficient semantic segmentation (e.g., larger feature maps and thin channel widths). To this end, we perform a comparison between the standard convolution and the widely-adopted depthwise separable convolution (SepConv) (Chollet, 2017;Howard et al., 2017;Tan and Le, 2019) in Figure 4. Here we set the shape of input feature map as 32 × 128 × 128, which is the most common shape in lightweight networks for semantic segmentation. The latency is evaluated by executing the block on a PC (i7-8700K/16GB RAM) with an 1070Ti GPU. The FLOPs-efficiency, which denotes the number of floating-point operations that is processed in per unit of run time, is utilized as the efficiency metric. As shown in Figure 4, the FLOPs-efficiency of standard convolution (3987 MFLOPs/ms) is about 10× more than that of SepConv (357 MFLOPs/ms). Accordingly, we utilize the standard convolution as the building block in LPS-Net.
The Number of Channels. The channel width is another important factor that influences the FLOPsefficiency of convolutions. Theoretically, the computational complexity necessitates quadratic progression with the increase of channel width. However, due to the highly-optimized software (e.g., BLAS, cuDNN) and hardware (e.g., CUDA Core, Tensor Core) in modern libraries/devices, the relation between defacto inference time and channel width in practice becomes fuzzy. To analyze this relation, we compare FLOPs and latency of 3 × 3 convolutions with different channel widths in Figure 5. As expected, FLOPs grows quadratically when increasing the channel width from 1 to 17 (see the blue curve in Figure 5(a)). Meanwhile, the latency only linearly grows and reaches plateaus at around 8 and 16 channel widths (see the red curve in Figure 5  setups of channel width with high FLOPs-efficiency are regarded as the "sweet spots" for convolutions. Similarly, high FLOPs-efficiencies are attained at 32 and 64 channel widths in Figure 5(b). These sweet points are mainly attributed to the highly parallelizable implementation of convolutions with 2 n -divisible channel width. As a result, we make the channel width of the convolution in LPS-Net to be 2 n -divisible with n as large as possible.

Multi-Path Interaction
In an effort to elegantly diffuse the complementary information across multiple scales, we design the multipath interaction module that encourages mutual exchange and interactions between pathes in our LPS-Net. Concretely, we define the interaction module I h , x out l } denote the input pair and output pair, respectively. The input pair consists of a high-resolution feature map x in h ∈ R C×H h ×W h and a low-resolution feature map x in l ∈ R C×H l ×W l , where H h ≥ H l and W h ≥ W l . The resolution of each path remains unchanged during the interaction.
Here, we study six variants of interaction modules which can be grouped into three categories ( Figure 6). For the first category (Direct-A/B), the transformed low-resolution features are directly integrated into the high-resolution ones through element-wise summation or channel-wise concatenation. Rather than simply combining the features from two paths, the second category employs attention mechanism to boost the multi-path interaction, namely Attention-A/B. Specifically, in Attention-A, an attention map Att(x in l ) is first calculated from x in l , and we apply it over the transformed x in h and x in l before feature aggregation: where F and U denote the convolutional transformation and spatial upsampling, respectively. Attention-B further upgrades the simple attention in a bi-direction manner by integrating each path with the attention derived from another path. In practice, the above four types of interaction modules are computationally expensive due to the heavy transformations and attentions. Thus, techniques (Chen et al., 2016;Tao et al., 2020;Yu et al., 2021a;Zhao et al., 2018) usually place them at the end of the multi-path networks, while leaving the path interactions at early stages unexploited. To alleviate this issue, two efficient interaction modules are devised to enable the interactions at early stages, namely Bilateral-A/B. Bilateral-A transforms low-resolution feature and injects it into the high-resolution one, and vice versa. Such symmetrical design treats each network path equally and the simple transformation triggers the interaction at early stages in an efficient fashion. Bilateral-B further eliminates the convolutional transformation in Bilateral-A, leading to the simplest interaction by solely applying spatial resizing and element-wise summation on input feature maps: where U and D denote upsampling and downsampling of the feature maps. The comparisons between all the six variants of interactions will be elaborated in the experiments, and we empirically verify that Bilateral-B achieves the best speed/accuracy tradeoff. Therefore, we adopt Bilateral-B as the default interaction module in our LPS-Net.

Scalable Architectures
Given the selected convolutional block and interaction module, a particular LPS-Net architecture can be for all δ ∈ ∆ do 14: end for 16: for all δ ∈ ∆ do 18: Expand N i−1 using δ with stepsize k δ . 21: P δ = P erf (N i δ ) 22: Train and evaluate N i δ for semantic segmentation. 23:

26: Select operation by comparing tradeoffs. 27:
Update: N i = N i−1 + k δ * · δ * , P = P δ * , L = L δ * 28: end while defined as N = {B, C, R}, where B, C, R are adjustable parameters described in Section 3.1. The network complexity is thus determined by these parameters from three dimensions. Depth dimension (B) is the number of blocks which determines the ability of the network to capture the high-level semantic information.
Width dimension (C), i.e., the number of channels at each stage, impacts the learning capacity of each convolution. Resolution dimension (R) represents the spatial granularity of each path. To balance the three dimensions when designing LPS-Net architecture, we start by building a tiny network, and then expand a single dimension at one time in a progressive manner.
Technically, we set the parameters of tiny network N 0 as B 0 = {1, 1, 1, 1, 1}, C 0 = {4, 8, 16, 32, 32}, and R 0 = {1/2, 0, 0}. Please note that the path with r = 0 is not included, and thus the tiny network only contains a single path. In this way, the tiny network is very efficient with only 0.38ms latency on a single GPU, and we further scale it to the heavier ones by multi-step expansion. All the candidates of expanding operations ∆ = ∆ B ∪ ∆ C ∪ ∆ R are defined in Table 1. When N = {B, C, R} is expanded by δ, δ will be added to N along its corresponding dimension and produce the expanded network (e.g. {B + δ, C, R} when δ ∈ ∆ B ). For each step, we greedily determine the executed expanding operation by comparing the tradeoff between speed and accuracy of each operation δ ∈ ∆. Specifically, for the i-th step, the target latency after expansion is first determined by the maximum latency across different expanding operations: where N i−1 denotes the architecture from the last step and Lat(·) measures the latency of the network. Then, a stepsize for each operation is calculated as: targeting for better aligning the resultant latency to L T after expanding different operations. Nevertheless, since the expansion only produces networks in discrete steps, the latencies of the expanded networks may exceed L T . If the selection of operation only depends on the performances of expanded networks, it may be biased to the operations which leads to a larger latency increase. To tackle this issue, the optimal expanding operation δ * is finally selected by maximizing the ratio of performance increase and latency increase after expanding at each step: where P erf (·) evaluates the performances of semantic segmentation. As such, starting from the tiny network N 0 , we iteratively perform the network expansion I times and finally derive a family of LPS-Net with different complexities. Algorithm 1 summarizes the expansion algorithm of LPS-Net.

Experiments
We empirically verify the merit of our LPS-Net by conducting a thorough evaluation of efficient semantic segmentation on Cityscapes (Cordts et al., 2016). Expressly, we first undertake the experiments for semantic segmentation to validate the lightweight designs of the convolutional blocks and the way of interactions across paths in LPS-Net with a pre-fixed structure. The second experiment examines the accuracy and latency tradeoffs along the progressive growth of network complexity of LPS-Net with regard to the number of convolutional blocks, the number of channels, and the input resolution. The third experiment compares three scalable levels of LPS-Net with several state-of-the-art fast semantic segmentation methods and demonstrates the better tradeoffs. Furthermore, we evaluate the transferability of the three versions of LPS-Net, which are learnt on Cityscapes, on BDD100K (Yu et al., 2018b) and CamVid (Brostow et al., 2008) datasets for semantic segmentation. Finally, we also migrate our LPS-Net to embedded devices and prove the efficacy on two different devices.

Datasets
The Cityscapes dataset contains 5,000 urban street images with high-quality pixel-level annotations of 19 classes. The image resolution is 1, 024 × 2, 048. All the images are split into three sets of 2,975, 500 and 1,525 for training, validation, and testing, respectively. For fair comparison, the additional set of 23,473 coarsely annotated images of this dataset is not utilized in our experiments. We train LPS-Net on the training set and evaluate the networks on the validation set. Following the standard protocol in (Fan et al., 2021;Lin et al., 2020;Zhang et al., 2019), LPS-Net is learnt on the training plus validation sets when submitting the model to online Cityscapes server and reporting the performance on the testing set.
To further examine the generalization of LPS-Net, we re-evaluate the networks on BDD100K (Yu et al., 2018b) and CamVid (Brostow et al., 2008) datasets. BDD100K is a recently released urban street dataset. For semantic segmentation, BDD100K includes 8,000 images being annotated with the consistent labels in Cityscapes. In between, 7,000 and 1,000 images are exploited for training and validation, respectively. The image resolution is 720 × 1, 280. CamVid consists of 5 video sequences with resolution 720×960, and is labeled at one frame per second with 11 semantic categories. 468 and 233 labeled frames in CamVid is utilized for training and testing, respectively.

Implementation Details
We implement our proposal on PyTorch (Paszke et al., 2019) platform and employ mini-batch Stochastic Gradient Descent (SGD) algorithm for model optimization.  (Ma et al., 2018), Inverted Residual (Sandler et al., 2018), Ghost Module (Han et al., 2020), and Residual/Bottleneck (He et al., 2016 In the training stage, we utilize the cross-entropy loss and train LPS-Net from scratch for 90K iterations, with a batch size of 16, momentum 0.9 and weight decay 0.0005. "Poly" policy with power 0.9 is adopted with initial learning rate 0.01. We use a crop size of 768×768 and exploit color-jittering, random horizontal flip and random scaling (0.5× ∼ 2.0×) to augment training data. Model re-parameterization (Ding et al., 2021) is also adopted for training. To further improve the capability of multi-path networks, we involve more powerful training strategies for the experiments in Section 4.5, including ImageNet (Russakovsky et al., 2015) pretraining, large crop size (768 × 1, 536), more training iterations (180K) and online hard element mining (Chen et al., 2020;Fan et al., 2021). Note that these strategies only enhance the network training and have no influence on model inference. Such training strategies employed for experiments on Cityscapes dataset are adjusted to train LPS-Net on BDD100K and CamVid datasets. For the experiments on BDD100K, we use a crop size of 720 × 1280 and the other strategies remain unchanged. For CamVid, we train the LPS-Net for 40K iterations, with a batch size of 12, an initial learning rate of 0.03, and a crop size 720 × 960. ImageNet pre-training and online hard element mining are also involved in experiments on BDD100K and CamVid.
In the inference stage, we input the image into our LPS-Net and perform semantic segmentation without any evaluation tricks (e.g., flipping, sliding-window testing and multi-scale inference), which in general improve the performance but at the expenses of extra latency. The Intersection over Union (IoU) per category and mean IoU over all the semantic categories are utilized as the performance metric. The inference latency of the network is measured by running networks with batch size of 1 on one NVIDIA GTX 1070Ti (Section 4.3 and Section 4.4), 1080Ti (Section 4.5) or two embedded devices (Section 4.6) with TensorRT in FP32 mode, unless otherwise stated.

Evaluation on Lightweight Designs
We first examine the lightweight designs of the convolutional blocks and the ways of interactions across paths in LPS-Net. We start the validation on the impact of the type of convolutional blocks. Specifically, we capitalize on seven convolutional blocks to build the corresponding multi-path networks. Following (Chen et al., 2016;Ding et al., 2021), B and R of all the networks are respectively set to {1, 2, 4, 7, 7} and {1, 1/4, 0}.
The channel widths C are set to {1c, 2c, 4c, 8c, 8c} for all the networks, where c is to adjust the inference latency of each network to approximate 5ms for fair comparisons. In order to purely verify the convolutional blocks irrespective of the interaction influence, we do not include any path interactions here and exploit channel-wise concatenation to aggregate the outputs from all paths of the multi-path network. Table 2 summarizes the mIoU performances, latencies, FLOPs, and FLOPs-efficiencies of the multi-path networks built on seven types of convolutional blocks. Similar to the observations in Section 3.2, 3×3 SepConv shows a very low FLOPs-efficiency. Conditioning on almost the same latency with 3×3 Conv, the utilization of 3 × 3 SepConv results in much fewer FLOPs, making the computational capability of the network weak. 3×3 SepConv is also inferior to 3 × 3 Conv in terms of mIoU performance. Furthermore, ShuffleNet Unit, Inverted Residual and Ghost Module are proven to be three very effective and lightweight convolutional blocks and all three blocks involve 3 × 3 SepConv. Inverted Residual and Ghost Module have benefited from the inverted residual structure with linear bottleneck and a series of linear transformations with cheap cost, respectively. They both greatly improve 3×3 SepConv, but the mIoU performances are still lower than that of 3 × 3 Conv. Residual contains a stack of two 3×3 convolutions with a shortcut connection and is extended to Bottleneck by stacking three convolutions (1 × 1, 3 × 3, 1 × 1) instead of two. The use of 1 × 1 convolution in Bottleneck leads to block fragmentation and affects its FLOPsefficiency. Compared to 3 × 3 Conv, an extra short Table 4: Network Parameters (Depth-B, Width-C and Resolution-R), latency, mIoU performance and expanding dimension of the network at each expansion step of LPS-Net.
Step B C R Latency (ms) mIoU (%) Dimension Name cut in Residual slightly lowers FLOPs-efficiency and increases the latency by ∼3%. Benefiting from the high FLOPs-efficiency and better tradeoff, 3×3 Conv exhibits the highest mIoU (66.2%) with the lowest latency (5.08ms). We further analyze the high FLOPsefficiency of 3×3 Conv from the perspective of devices. FLOPs-lightweight convolutions (e.g. Ghost Module and 3 × 3 SepConv) usually suffer from a low ratio of computing to memory operations (Gholami et al., 2018) and introduce extra overheads like kernel launching and synchronization. In contrast, 3×3 Conv is more friendly to devices and could be accelerated by the Winograd supported in modern computing libraries like cuDNN. As such, 3×3 Conv has high FLOPs-efficiency. Overall, 3×3 Conv achieves the best tradeoff and thus we exploit it as the convolutional block in LPS-Net.
Then, we study how the interactions across paths influence the accuracy-latency tradeoff of the multipath networks. In this comparison, the network parameters B and R are kept unchanged. We fix c = 16 and exploit 3 × 3 Conv as the convolutional block to construct the multi-path networks. Table 3 details the mIoU performance and latency when integrating different path interactions in multi-path networks. As expected, the results across six ways of interactions consistently indicate that using interaction exhibits better mIoU performance and higher latency against Baseline which does not involve path interaction. More specifically, the larger mIoU improvements are attained when employing the Bilateral-A/B interactions than the Direct-A/B and Attention-A/B interactions. Bilateral-B purely basing on bilinear interpolation leads to less increase of latency than Bilateral-A which uses two additional convolutional transformations. Among all the six interactions, Bilateral-B achieves the largest mIoU gain of 3.0% with the least extra latency of 0.29ms, and hence we utilize Bilateral-B as the path interaction in our LPS-Net.

Evaluation on Progressively-Scalable Scheme
Next, we analyze the accuracy-latency balance as the progressive expansion of LPS-Net proceeds. Figure 7 depicts the first 14 steps of the process on Cityscapes. Table 4 further details the parameters, latency, mIoU performance and expanding dimension of the network at each expansion step of LPS-Net in Figure 7. We initialize LPS-Net with a tiny structure and the network parameters are B 0 = {1, 1, 1, 1, 1}, C 0 = {4, 8, 16, 32, 32}, R 0 = {1/2, 0, 0}. The mIoU and latency of such start point is 24.1% and 0.38ms, respectively. As shown in the figure, expanding one  (Li et al., 2019a) 1024 × 1024 -71.3 100.0 TitanX DFANet B (Li et al., 2019a) 1024 × 1024 -67.1 120.0 TitanX DFANet A' (Li et al., 2019a) 512 × 1024 -70.3 160.0 TitanX Fast-SCNN (Poudel et al., 2019) 1024 × 1024 69.2 68.0 123.5 TitanXp MSFNet (Si et al., 2020) 512 × 1024 -71.3 117.0 2080Ti SFNet(DF1) (Li et al., 2020b) 1024 × 2048 -74.5 121.0 1080Ti TensorRT STDC1Seg-50 (Fan et al., 2021) 512 × 1024 72.2 71.9 250.4 1080Ti TensorRT STDC2Seg-50 (Fan et al., 2021) 512 certain dimension of Depth (the number of convolutional blocks), Width (the number of channels), or the input Resolution in LPS-Net always improves the mIoU performance. The fact that the selected dimension is various to strike the best mIoU performance-inference latency tradeoff at different steps necessitates the use of multiple dimensions for expansion. The first step is along the network Depth direction and significantly boosts up the mIoU to 41.4% with only 0.11ms additional latency. We speculate that this may be the result of feature enrichment by stacking more blocks particularly in the context of small channel width and low image resolution. In this case, deepening networks could readily enhance the network capability. The second and third steps further expand Width dimension, and the network after three steps achieves 57.8% mIoU with less than 1ms latency. The results indicate that the network tends to expand the Depth and Width dimensions especially at the very early steps. At the sixth step, the expansion proceeds along the Resolution direction and starts to develop a two-path networks. The next two consecutive steps also work on the factor of Resolution and increase R to {1, 1/4, 0}. The mIoU performance of the two-path networks at the eighth step reaches 69.5% and the latency is 5.17ms. That basically verifies the effectiveness of multi-path design to manage the accuracy-latency balance for semantic segmentation. The following steps continue the expansion along either the Width or Resolution dimension. Such an expansion process indicates some practical insights for efficient semantic segmentation. For example, stacking more blocks for tiny networks could effectively improve the segmentation with negligible extra latency. The multipath fusion is not always the best choice especially when the networks' inference latency is extremely constrained (e.g., ≤ 2ms). Compared to increasing channel width and enlarging input resolutions, fusing three or more paths is not optimal for efficient semantic segmentation.
To cover different levels of LPS-Net in complexity, we choose the networks at the seventh, eighth, and tenth steps as the small, medium, and large version of LPS-Net. We name these networks as LPS-Net-S, LPS-Net-M, and LPS-Net-L, respectively.

Comparisons with State-of-the-Art Methods
We compare our LPS-Net with several state-of-the-art fast semantic segmentation methods. To ensure the inference speed is comparable with the most recent works (Chen et al., 2020;Fan et al., 2021;Yu et al., 2021a), we report the inference FPS of LPS-Net on an NVIDIA GTX 1080Ti GPU card here. Table 5 summarizes the comparisons of mIoU performance and inference FPS on Cityscapes dataset. Note that only the models with FPS≥100 are included in the table. Overall, the results across validation and testing sets consistently indicate that LPS-Net exhibits better tradeoffs than other techniques including hand-crafted networks (e.g., (Fan et al., 2021;Li et al., 2019)) and auto-designed models (e.g., (Chen et al., 2020;Zhang et al., 2019)). In particular, the mIoU score of LPS-Net-S achieves 73.4% on test set, making the absolute improvement over the best competitor STDC1Seg-50 by 1.5%. Moreover, LPS-Net-S is faster than STDC1Seg-50 and the inference speed reaches 413.5 FPS, which is impressive. The large version of LPS-Net, i.e., LPS-Net-L, attains 78.2%/77.3% mIoU on validation/testing set at 151.8 FPS. It runs 49% faster than DDRNet23slim with comparable mIoU. This confirms the effectiveness of LPS-Net. We showcase the qualitative results of LPS-Net-S, LPS-Net-M, and LPS-Net-L in Figure 8. Similar to the observation in Table 5, the quality of prediction is consistently improved when the LPS-Net is incrementally expanded. For example, the prediction in the regions of pole (first row), car (second row), and wall (third row) is amended by expanding the complexity of LPS-Net.

LPS-Net-S on Embedded Devices
The real-world scenarios often demand the deployments on embedded devices. Here, we also test the small version of our LPS-Net, i.e., LPS-Net-S, on two devices of a single Nano and a single TX2, both of which are based on NVIDIA Jetson platform. Table 8 compares LPS-Net-S with DF1-Seg-d8 (Li et al., 2019b), FasterSeg (Chen et al., 2020), BiSeNetV2 (Yu et al., 2018a), and STDC1Seg-50 (Fan et al., 2021), regarding the mIoU performance on Cityscapes and inference  speed (FPS) on the two devices. Note that the inference speed is measured by using FP32 data precision to avoid performance degradation. Clearly, LPS-Net-S strikes the superior tradeoff and achieves 10.0/25.6 FPS on Nano/TX2 at resolution 1024×2048, which is 41%/50% faster than the best competitor STDC1Seg-50 (Fan et al., 2021). The results demonstrate the advantage of LPS-Net on embedded devices.

Conclusion and Discussion
We have presented Lightweight and Progressively-Scalable Networks (LPS-Net) which explores the economic design and progressively scales up the network for efficient semantic segmentation. Particularly, we analyze the basic convolutional block and the way of path interaction in multi-path framework which could affect the accuracy-latency tradeoff for semantic segmentation. Empirically, we suggest to use 3 × 3 Conv in convolutional blocks and bilinear interpolation to implement interactions across paths. Based on these guidelines, we first build a tiny network and then extends the tiny network to a series of larger ones through expanding a certain dimension of Width, Depth or Resolution at one time. Experiments conducted on three datasets, i.e., Cityscapes, CamVid and BDD100K, validate our proposal and analysis on both GPUs and embedded devices. More remarkably, our LPS-Net shows impressive accuracy-latency tradeoff on Cityscapes: 413.5 FPS on an NVIDIA GTX 1080Ti GPU or 25.6 FPS on an NVIDIA TX2, with mIoU of 73.4% on testing set.