Learning a Discriminative Feature Network for Semantic Segmentation

Most existing methods of semantic segmentation still suffer from two aspects of challenges: intra-class inconsistency and inter-class indistinction. To tackle these two problems, we propose a Discriminative Feature Network (DFN), which contains two sub-networks: Smooth Network and Border Network. Specifically, to handle the intra-class inconsistency problem, we specially design a Smooth Network with Channel Attention Block and global average pooling to select the more discriminative features. Furthermore, we propose a Border Network to make the bilateral features of boundary distinguishable with deep semantic boundary supervision. Based on our proposed DFN, we achieve state-of-the-art performance 86.2% mean IOU on PASCAL VOC 2012 and 80.3% mean IOU on Cityscapes dataset.


Introduction
Semantic segmentation is a fundamental technique for numerous computer vision applications like scene understanding, human parsing and autonomous driving. With the recent development of the convolutional neural network, especially the Fully Convolutional Network (FCN) [27], a lot of great work such as [40,6,19,30] have obtained promising results on the benchmarks. However, the features learned by these methods are usually not discriminative to differentiate 1) the patches which share the same semantic label but different appearances, named intra-class inconsistency as shown in the first row of Figure 1; 2) the two adjacent patches which have different semantic labels but with similar appearances, named inter-class indistinction as shown in the second row of Figure 1.
To address these two challenges, we rethink the semantic segmentation task from a more macroscopic point of view. In this way, we regard the semantic segmentation as Figure 1. Hard examples in semantic segmentation. The second column is the output of FCN based model. The third column is the output of our proposed approach. In the first row, the left bottom corner of the cow is recognized as a horse. This is the Intra-class Inconsistency problem. In the second row, the computer case has the similar blue light and black shell with the computer screen, which is hard to distinguish. This is the Inter-class Indistinction problem. a task to assign a consistent semantic label to a category of things, rather than to each single pixel. From a macroscopic perspective, regarding each category of pixels as a whole, inherently considers both intra-class consistency and interclass variation. It means that the task demands discriminative features. To this end, we present a novel Discriminative Feature Network (DFN) to learn the feature representation which considers both the "intra-class consistency" and the "inter-class distinction".
Our DFN involves two components: Smooth Network and Border Network, as Figure 2 illustrates. The Smooth Network is designed to address the intra-class inconsistency issue. To learn a robust feature representation for intra-class consistency, we usually consider two crucial factors. On the one hand, we need multi-scale and global context features to encode the local and global information. For example, the small white patch only in Figure 1(a) usually cannot predict the correct category due to the lack of sufficient context information. On the other hand, as multi-scale context is introduced, for a certain scale of thing, the features have different extent of discrimination, some of which may predict a false label. Therefore, it is necessary to select the discriminative and effective features. Motivated by these two aspects, our Smooth Network is presented based on the U-shape [30,19,31,11,36] structure to capture the multi-scale context information, with the global average pooling [21,24,40,6] to capture the global context. Also, we propose a Channel Attention Block (CAB), which utilizes the high-level features to guide the selection of lowlevel features stage-by-stage.
Border Network, on the other hand, tries to differentiate the adjacent patches with similar appearances but different semantic labels. Most of the existing approaches [24,40,6,30] consider the semantic segmentation task as a dense recognition problem, which usually ignores explicitly modeling the inter-class relationship. Consider the example in Figure 1(d), if more and more global context is integrated into the classificiation process, the computer case next to the monitor can be easily misclassified as a monitor due to the similar appearance. Thus, it is significant to explicitly involve the semantic boundary to guide the learning of the features. It can amplify the variation of features on both sides. In our Border Network, we integrate semantic boundary loss during the training process to learn the discriminative features to enlarge the "inter-class distinction".
In summary, there are four contributions in our paper: • We rethink the semantic segmentation task from a new macroscopic point of view. We regard the semantic segmentation as a task to assign a consistent semantic label to one category of things, not just at the pixel level. • We propose a Discriminative Feature Network to simultaneously address the "intra-class consistency" and "inter-class variation" issues. Experiments on PAS-CAL VOC 2012 and Cityscapes datasets validate the effectiveness of our proposed algorithm. • We present a Smooth Network to enhance the intraclass consistency with the global context and the Channel Attention Block. • We design a bottom-up Border Network with deep supervision to enlarge the variation of features on both sides of the semantic boundary. This can also refine the semantic boundary of prediction.

Related Work
Recently, lots of approaches based on FCN have achieved high performance on different benchmarks [42,9,8]. Most of them are still constrained by intra-class inconsistency and inter-class indistinction issues.
Encoder-Decoder: The FCN model has inherently encoded different levels of feature. Naturally, some methods integrate them to refine the final prediction. This branch of methods mainly consider how to recover the reduced spatial information caused by consecutive pooling operator or convolution with stride. For example, SegNet [1] utilizes the saved pool indices to recover the reduced spatial information. U-net [31] uses the skip connection, while the Global Convolutional Network [30] adapts the large kernel size. Besides, LRR [11] adds the Laplacian Pyramid Reconstruction network, while RefineNet [19] utilizes multipath refinement network. However, this type of architecture ignores the global context. In addition, most methods of this type are just summed up the features of adjacent stages without consideration of their diverse representation. This leads to some inconsistent results.
Global Context: Some modern methods have proven the effectiveness of global average pooling. ParseNet [24] firstly applies global average pooling in the semantic segmentation task. Then PSPNet [40] and Deeplab v3 [6] respectively extend it to the Spatial Pyramid Pooling [13] and Atrous Spatial Pyramid Pooling [5], resulting in great performance in different benchmarks. However, to take advantage of the pyramid pooling module sufficiently, these two methods adopt the base feature network to 8 times downsample with atrous convolution [5,38], which is timeconsuming and memory intensive.
Attention Module: Attention is helpful to focus on what we want. Recently, the attention module becomes increasingly a powerful tool for deep neural networks [28,33,16,3]. The method in [7] pays attention to different scale information. In this work, we utilize channel attention to select the features similar to SENet [16].
Semantic Boundary Detection: Boundary detection is a fundamental challenge in computer vision. There are lots of specific methods proposed for the task of boundary detection [39,36,37,25]. Most of these methods straightly concatenate the different level of features to extract the boundary. However, in this work, our goal is to obtain the features with inter-class distinction as much as possible with accurate boundary supervision. Therefore, we design a bottomup structure to optimize the features on each stage.

Method
In this section, we first detailedly introduce our proposed Discriminative Feature Network containing Smooth Network and Border Network. Then, we elaborate how these two networks specifically handle the intra-class consistency issue and the inter-class distinction issue. Finally, we describe the complete encoder-decoder network architecture, Discriminative Feature Network.

Smooth network
In the task of semantic segmentation, most of modern methods consider it as a dense prediction issue. However, the prediction sometimes has incorrect results in some parts, especially the parts of large regions and complex scenes, which is named intra-class inconsistency issue.
The intra-class inconsistency problem is mainly due to the lack of context. Therefore, we introduce the global context with global average pooling [24,21,40,6]. However, global context just has the high semantic information, which is not helpful for recovering the spatial information. Consequently, we further need the multi-scale receptive view and context to refine the spatial information, as most modern approaches [40,6,30] do. Nevertheless, there exists a problem that the different scales of receptive views produce the features with different extents of discrimination, leading to inconsistent results. Therefore, we need to select more discriminative features to predict the unified semantic label of one certain category.
In our proposed network, we use ResNet [14] as a base recognition model. This model can be divided into five stages according to the size of the feature maps. According to our observation, the different stages have different recognition abilities resulting in diverse consistency manifestation. In the lower stage, the network encodes finer spatial information, however, it has poor semantic consistency because of its small receptive view and without the guidance of spatial context. While in the high stage, it has strong semantic consistency due to large receptive view, however, the prediction is spatially coarse. Overall, the lower stage makes more accurate spatial predictions, while the higher stage gives more accurate semantic predictions. Based on this observation, to combine their advantages, we propose a Smooth Network to utilize the high stage's consistency to guide the low stage for the optimal prediction.
We observe that in the current prevalent semantic segmentation architecture, there are mainly two styles. The first one is "Backbone-Style", such as PSPNet [40], Deeplab v3 [6]. It embeds different scale context information to improve the consistency of network with the Pyramid Spatial Pooling module [13] or Atrous Spatial Pyramid Pooling module [5]. The other one is "Encoder-Decoder-Style", like RefineNet [19], Global Convolutional Network [30]. This style of network utilizes the inherent multi-scale context of different stage, but it lacks the global context which has the strongest consistency. In addition, when the network combines the features of adjacent stages, it just sums up these features by channel. This operation ignores the diverse consistency in different stages. To remedy the defect, we first embed a global average pooling layer [24] to extend the U-shape architecture [27,36] to a Vshape architecture. With the global average pooling layer, we introduce the strongest consistency constraint into the network as a guidance. Furthermore, to enhance consistency, we design a Channel Attention Block, as shown in Figure 2(c). This design combines the features of adjacent stages to compute a channel attention vector 3(b). The fea-  In (a), the yellow block represents the feature of low stage, while the red one represents high stage. We concatenate the features of adjacent stages to compute a weight vector, which re-weights the feature maps of low stage. The hotter color represents the high weight value. In (b), it is the real attention value vector from the stage-4 channel attention block. The deeper blue represents the higher weight value.
tures of high stage provide a strong consistency guidance, while the features of low stage give the different discrimination information of features. In this way, the channel attention vector can select the discriminative features.
Channel attention block: Our Channel Attention Block (CAB) is designed to change the weights of the features on each stage to enhance the consistency, as illustrated in Figure 3. In the FCN architecture, the convolution operator outputs a score map, which gives the probability of each class at each pixel. In Equation 1, the final score at score map is just summed over all channels of feature maps.
where x is the output feature of network. w represents the convolution kernel. And k ∈ {1, 2, . . . , K}. K is the number of channels. D is the set of pixel positions.
where δ is the prediction probability. y is the output of network.
As shown in Equation 1 and Equation 2, the final predicted label is the category with highest probability. Therefore, we assume that the prediction result is y 0 of a certain patch, while its true label is y 1 . Consequently, we can introduce a parameter α to change the highest probability value from y 0 to y 1 , as Equation 3 shows.
whereȳ is the new prediction of network and α = Sigmoid(x; w) Based on the above formulation of the Channel Attention Block (CAB), we can explore its practical significance. In Equation 1, it implicitly indicates that the weights of different channels are equal. However, as mentioned in Section 1, the features in different stages have different degrees of discrimination, which results in different consistency of prediction. In order to obtain the intra-class consistent prediction, we should extract the discriminative features and inhibit the indiscriminative features. Therefore, in Equation 3, the α value applies on the feature maps x, which represents the feature selection with CAB. With this design, we can make the network to obtain discriminative features stage-wise to make the prediction intra-class consistent.
Refinement residual block: The feature maps of each stage in feature network all go through the Refinement Residual Block, schematically depicted in Figure 2(b). The first component of the block is a 1 × 1 convolution layer. We use it to unify the number of channels to 512. Meanwhile, it can combine the information across all channels. Then the following is a basic residual block, which can refine the feature map. Furthermore, this block can strengthen the recognition ability of each stage, inspired from the architecture of ResNet [14,15].

Border network
In the semantic segmentation task, the prediction is confused with the different categories with similar appearances, especially when they are adjacent spatially. Therefore, we need to amplify the distinction of features. With this motivation, we adopt a semantic boundary to guide the feature learning. To extract the accurate semantic boundary, we apply the explicit supervision of semantic boundary, which makes the network learn a feature with strong inter-class distinctive ability. Therefore, we propose a Border Network to enlarge the inter-class distinction of features. It directly learns a semantic boundary with an explicit semantic boundary supervision, similar to a semantic boundary detection task. This makes the features on both sides of semantic boundary distinguishable.
As stated in Section 3.1, the feature network has different stages. The low stage features have more detailed information, while the high stage features have higher semantic information. In our work, we need semantic boundary with more semantic meanings. Therefore, we design a bottomup Border Network. This network can simultaneously get accurate edge information from low stage and obtain semantic information from high stage, which eliminates some original edges lack of semantic information. In this way, the semantic information of high stage can refine the detailed edge information from low stage stage-wise. The supervisory signal of the network is obtained from the semantic segmentation's groundtruth with a traditional image processing method, such as Canny [2].
To remedy the imbalance of the positive and negative samples, we use focal loss [22] to supervise the output of the Border Network, as shown in Equation 4. We adjust the parameters α and γ of focal loss for better performance.
where p k is the estimated probability for class k, k ∈ {1, 2, . . . , K}. And K is the maximum value of class label.
The Border Network mainly focuses on the semantic boundary which separates the classes on two sides of the boundary. For extracting accurate semantic boundary, the features on both sides will become more distinguishable. This exactly reaches our goal to make the features with inter-class distinction as much as possible.

Network Architecture
With Smooth Network and Border Network, we propose our Discriminative Feature Network for semantic segmentation as illustrated in Figure 2 (a).
We use pre-trained ResNet [14] as a base network. In the Smooth Network, we add the global average pooling layer on the top of the network to get the strongest consistency. Then we utilize the channel attention block to change the weights of channels to further enhance the consistency. Meanwhile, in the Border Network, with the explicit semantic boundary supervision, the network obtains accurate semantic boundary and makes the bilateral features more distinct. With the support of both sub-networks, the intra-class features become more consistent, while the inter-class ones grow more distinct.
For explicit feature refinement, we use deep supervision to get better performance and make the network easier to optimize. In the Smooth Network, we use the softmax loss to supervise the each stage's upsampled output excluding the global average pooling layer, while we use the focal loss to supervise the outputs of Border Network. Finally, we use a parameter λ to balance the segmentation loss s and the boundary loss b , as Equation 7 shows. s = Sof tmaxLoss(y; w)

Experimental Results
We evaluate our approach on two public datasets: PAS-CAL VOC 2012 [9] and Cityscapes [8]. We first introduce the datasets and report the implementation details. Then we evaluate each component of the proposed method, and analyze the results in detail. Finally, we present the comparison results with other state-of-the-art methods.
PASCAL VOC 2012: The PASCAL VOC 2012 is a wellknown semantic segmentation benchmark which contains 20 object classes and one background, involving 1,464 images for training, 14,449 images for validation and 1,456 images for testing. The original dataset is augmented by the Semantic Boundaries Dataset [12], resulting in 10,582 images for training.

Implementation details
Our proposed network is based on the ResNet-101 pretrained on ImageNet [32]. And we use the FCN4 [27,36] as our base segmentation framework.
Training: We train the network using mini-batch stochastic gradient descent (SGD) [17] with batch size 32, momentum 0.9 and weight decay 0.0001. Inspired by [5,24], we use the "poly" learning rate policy where the learning rate is multiplied by 1 − iter max iter power with power 0.9 and initial learning rate 4e −3 . As for the λ, we finally use the value of 0.1 after a series of comparison experiments. For measuring the performance of our proposed network, we use the mean pixel intersection-over-union (mean IOU) as the metric.
Data augmentation: We use mean subtraction and random horizontal flip in training for both PASCAL VOC 2012 and Cityscapes. In addition, we find it is crucial to randomly scale the input images, which improves the performance obviously. We use 5 scales {0.5, 0.75, 1, 1.5, 1.75} on both datasets.

Ablation study
In this subsection, we will step-wise decompose our approach to reveal the effect of each component. In the following experiments, we evaluate all comparisons on PAS-CAL VOC 2012 dataset [9]. And we report the comparison results in PASCAL VOC 2012 dataset [9] and Cityscapes dataset [8].

Smooth network
We use the ResNet-101 as our base feature network, and directly upsample the ouput. First, we evaluate the performance of the base ResNet-101, as shown in Table 1. Then we extend the base network to FCN4 structure [27,36] with our proposed Refinement Residual Block (RRB), which improves the performance from 72.86% to 76.65%, as Table 2 shows. We visualize the effect of the Smooth Network. Figure 4 presents some examples of semantic segmentation results. Obviously, our Smooth Network can effectively make the prediction more consistent.
Ablation for global pooling: We need the features with strong consistency. Thus based our observation in Section 3, we add the global average pooling on the top of the network. As shown in Table 2, the global average pooling introduces the strongest consistency to guide other stages. This improves the performance from 76.65% to 78.20%, which is an obvious improvement.
Ablation for deep supervision: To refine the hierarchical features, we use deep supervision. We add the softmax loss on each stage excluding the global average pooling layer. As shown in Table 2, this further improves the performance by almost 0.4%.
Ablation for channel attention block: Based on the aforementioned architecture, we add the Channel Attention Block (CAB). It utilizes the high stage to guide the  low stage with a channel attention vector to enhance consistency, which improves the performance from 78.51% to 79.54% over evaluation, as Table 2 shows.

Border network
While the Smooth Network pays attention to the intra-class consistency, the Border Network focuses on the inter-class indistinction. Due to the accurate boundary supervisory signal, the network amplifies the distinction of bilateral feature to extract the semantic boundary. Then we integrate the Border Network into the Smooth Network. This improves the performance from 79.54% to 79.67%, as shown in Table 3. The Border Network optimizes the semantic boundary, which is a comparably small part of the whole image, so this design makes a minor improvement. We visualize the effect of Border Network, as shown in Figure 5. In addition, Figure 6 shows the predicted semantic boundary of Border Network. We can obviously observe that the Border Network can focus on the semantic boundary preferably.   Figure 7. Example results of DFN in the stage-wise refinement process on PASCAL VOC 2012 dataset. The first column is the original image and groundtruth. The last is the refinement process of two networks. The segmentation prediction in lower stage is more spatial coarse, and the higher is finer. While the boundary prediction in lower stage contains more edges not belong to semantic boundary, the semantic boundary in higher stage is more pure. stage-wise mechanism, the Smooth Network and Border Network respectively refine the segmentation and boundary prediction, as shown in Figure 7. The gradually accurate predictions validate the effectiveness of the stage-wise mechanism.
Performance evaluation on Cityscapes: We also evaluate our approach on the Cityscapes dataset [8]. In training, our crop size of image is 800 × 800. We observe that for the high resolution of image the large crop size is useful. The test performance results are specifically reported in Table 6. We visualize the results of our approach on the Cityscapes dataset, as shown in Figure 9.  Table 6. Performance on Cityscapes test set. The "-" indicates that the method do not present this result in its paper.

Conclusion
We redefine the semantic segmentation from a macroscopic view of point, regarding it as a task to assign a consistent semantic label to one category of objects, rather than to each single pixel. Inherently, this task requires the intra-class consistency and inter-class distinction. Aiming to consider both sides, we propose a Discriminative Feature Network, which contains two sub-networks: Smooth Network and Border Network. With the bidirectional stagewise mechanism, our approach can capture the discriminative features for semantic segmentation. Our experimental results show that the proposed approach can significantly improve the performance on the PASCAL VOC 2012 and Cityscapes benchmarks.