Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> VI. SCENE LABELING <s> We adopt Convolutional Neural Networks (CNN) as our parametric model to learn discriminative features and classifiers for local patch classification. As visually similar pixels are indistinguishable from local context, we alleviate such ambiguity by introducing a global scene constraint. We estimate the global potential in a non-parametric framework. Furthermore, a large margin based CNN metric learning method is proposed for better global potential estimation. The final pixel class prediction is performed by integrating local and global beliefs. Even without any post-processing, we achieve state-of-the-art performance on SiftFlow and competitive results on Stanford Background benchmark. <s> BIB001 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> VI. SCENE LABELING <s> Most existing approaches for RGB-D indoor scene labeling employ hand-crafted features for each modality independently and combine them in a heuristic manner. There has been some attempt on directly learning features from raw RGB-D data, but the performance is not satisfactory. In this paper, we propose an unsupervised joint feature learning and encoding (JFLE) framework for RGB-D scene labeling. The main novelty of our learning framework lies in the joint optimization of feature learning and feature encoding in a coherent way, which significantly boosts the performance. By stacking basic learning structure, higher level features are derived and combined with lower level features for better representing RGB-D data. Moreover, to explore the nonlinear intrinsic characteristic of data, we further propose a more general joint deep feature learning and encoding (JDFLE) framework that introduces the nonlinear mapping into JFLE. The experimental results on the benchmark NYU depth dataset show that our approaches achieve competitive performance, compared with the state-of-the-art methods, while our methods do not need complex feature handcrafting and feature combination and can be easily applied to other data sets. <s> BIB002 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> VI. SCENE LABELING <s> In image labeling, local representations for image units are usually generated from their surrounding image patches, thus long-range contextual information is not effectively encoded. In this paper, we introduce recurrent neural networks (RNNs) to address this issue. Specifically, directed acyclic graph RNNs (DAG-RNNs) are proposed to process DAG-structured images, which enables the network to model long-range semantic dependencies among image units. Our DAG-RNNs are capable of tremendously enhancing the discriminative power of local representations, which significantly benefits the local classification. Meanwhile, we propose a novel class weighting function that attends to rare classes, which phenomenally boosts the recognition accuracy for non-frequent classes. Integrating with convolution and deconvolution layers, our DAG-RNNs achieve new state-of-the-art results on the challenging SiftFlow, CamVid and Barcelona benchmarks. <s> BIB003
Scene labeling aims to assign one of a set of semantic labels to each pixel of a scene image. It is very challenging due to the fact that some classes may be indistinguishable in a close-up view. Generally, "thing" pixels (cars, person, etc) in real world images can be quite different due to their scale, illumination and pose variation. Recently, deep learning based methods have achieved very promising results for scene labeling. We review some of them as follows. Shuai et al. BIB001 propose to adopt Convolutional Neural Networks (CNNs) as a parametric model to learn discriminative features and classifier for scene labeling. Figure 16 illustrates the framework of the proposed method. First, global scene semantics are used to remove the ambiguity of local context by transferring class dependencies and priors from similar exemplars. Then, the global potential is decoupled to the aggregation of global beliefs over pixels. The labeling result can be obtained by integrating the local and global beliefs. Finally, a large margin based metric learning is introduced to make the estimation of global belief more accurate. As a result, the proposed model is able to achieve state-of-the-art results on the SiftFlow benchmark and very competitive results on the Stanford Background dataset. Shuai et al. BIB003 present a directed acyclic graph RNNs (DAG-RNNs) for scene labeling to model long-range semantic dependencies among image units. Figure 17 illustrates the Fig. 18 . Illustration of the framework for RGB-D indoor scene labeling in the approach BIB002 . architecture of the full labeling network in the proposed method. First, undirected cyclic graphs (UCG) are adopted to model interactions among image units. Due to the loopy property of UCGs, RNNs are not directly applicable to UCGstructured images. Thus, the UCG is decomposed to several directed acyclic graphs (DAGs). Then, each hidden layer is generated independently through applying DAG-RNNs to the corresponding DAG-structured image, and they are integrated to produce the context-aware feature maps. As a result, the local representations are able to embed the abstract gist of the image, so their discriminative power are enhanced remarkably. It is reported that the DAG-RNNs achieve new state-of-the-art results on the challenging SiftFlow, CamVid and Barcelona benchmarks. Wang et al. BIB002 develop an unsupervised joint feature learning and encoding (JFLE) framework for RGB-D scene labeling. Figure 18 illustrates the framework for RGB-D indoor scene labeling in the proposed method. First, feature learning and encoding are jointly performed in a two-layer stacked structure, called joint feature learning and encoding framework (JFLE). To further extend the JFLE framework to a more general framework called joint deep feature learning and encoding (JDFLE), a deep model is used with stacked nonlinear layers to model the input data. The input to the learning structure (either JFLE or JDFLE) is a set of patches densely sampled from RGB-D images, and the learning output is the set of corresponding path features, which are then combined to generate superpixel features. Finally, linear SVMs are trained to map superpixel features to scene labels. It is reported that the proposed feature learning framework can achieve competitive performance on the benchmark NYU depth dataset. Scene labeling can be used to understand urban images captured from surveillance cameras. It is a very important component of smart city, in which city elements such as "road" and "building" are required to be recognized. Deep models can leverage on big data from smart city to learn hierarchical features for labeling each pixel of scene images.