To improve segmentation performance, a novel neural network architecture (termed DFCN-DCRF) is proposed, which combines an RGB-D fully convolutional neural network (DFCN) with a depth-sensitive fully-connected conditional random field (DCRF). First, a DFCN architecture which fuses depth information into the early layers and applies dilated convolution for later contextual reasoning is designed. Then, a depth-sensitive fully-connected conditional random field (DCRF) is proposed and combined with the previous DFCN to refine the preliminary result. Comparative experiments show that the proposed DFCN-DCRF achieves competitive performance compared with state-of-the-art methods.
Based on some labeled RGB-D image datasets @cite_4 @cite_20 @cite_3 @cite_2 , many studies have tried to incorporate the depth information to generate a better performance. @cite_21 shows that simply stack the depth with RGB as a 4-channels input cannot improve the performance in a significant way. @cite_25 presented a different representation of depth information referred as HHA. It comprised horizontal disparity, height from the ground and the angle between the local surface normal and gravity direction, and got a good results @cite_21 @cite_25 . However, @cite_14 . argued that the HHA representation did not contain more information than the raw depth itself, and it required a high computation cost. In their work, a fusion-based CNN architecture was presented. The network consisted of two branches of encoding networks, i.e., a depth branch and an RGB branch. The feature representation in these two branches was then fused into the master branch. There are two ways of fusing approaches, i.e., sparse fusion and dense fusion. It was proved that the sparse one is better. Therefore, in our work, we fuse RGB and depth channel feature representation in a sparse way from Conv1 to Conv4.
@cite_14 @cite_4 @cite_21 @cite_3 @cite_2 @cite_25 @cite_20
2587989515 2416791088 2489780108 2793381735
In this paper we address the problem of semantic labeling of indoor scenes on RGB-D data. With the availability of RGB-D cameras, it is expected that additional depth measurement will improve the accuracy. Here we investigate a solution how to incorporate complementary depth information into a semantic segmentation framework by making use of convolutional neural networks (CNNs). Recently encoder-decoder type fully convolutional CNN architectures have achieved a great success in the field of semantic segmentation. Motivated by this observation we propose an encoder-decoder type network, where the encoder part is composed of two branches of networks that simultaneously extract features from RGB and depth images and fuse depth features into the RGB feature maps as the network goes deeper. Comprehensive experimental evaluations demonstrate that the proposed fusion-based architecture achieves competitive results with the state-of-the-art methods on the challenging SUN RGB-D benchmark obtaining 76.27 global accuracy, 48.30 average class accuracy and 37.29 average intersection-over-union score. In this paper, we explore various aspects of fusing LIDAR and color imagery for pedestrian detection in the context of convolutional neural networks (CNNs), which have recently become state-of-art for many vision problems. We incorporate LIDAR by up-sampling the point cloud to a dense depth map and then extracting three features representing different aspects of the 3D scene. We then use those features as extra image channels. Specifically, we leverage recent work on HHA [9] (horizontal disparity, height above ground, and angle) representations, adapting the code to work on up-sampled LIDAR rather than Microsoft Kinect depth maps. We show, for the first time, that such a representation is applicable to up-sampled LIDAR data, despite its sparsity. Since CNNs learn a deep hierarchy of feature representations, we then explore the question: At what level of representation should we fuse this additional information with the original RGB image channels? We use the KITTI pedestrian detection dataset for our exploration. We first replicate the finding that region-CNNs (R-CNNs) [8] can outperform the original proposal mechanism using only RGB images, but only if fine-tuning is employed. Then, we show that: 1) using HHA features and RGB images performs better than RGB-only, even without any fine-tuning using large RGB web data, 2) fusing RGB and HHA achieves the strongest results if done late, but, under a parameter or computational budget, is best done at the early to middle layers of the hierarchical representation, which tend to represent midlevel features rather than low (e.g. edges) or high (e.g. object class decision) level features, 3) some of the less successful methods have the most parameters, indicating that increased classification accuracy is not simply a function of increased capacity in the neural network. Semantic labeling of RGB-D scenes is crucial to many intelligent applications including perceptual robotics. It generates pixelwise and fine-grained label maps from simultaneously sensed photometric (RGB) and depth channels. This paper addresses this problem by (i) developing a novel Long Short-Term Memorized Context Fusion (LSTM-CF) Model that captures and fuses contextual information from multiple channels of photometric and depth data, and (ii) incorporating this model into deep convolutional neural networks (CNNs) for end-to-end training. Specifically, contexts in photometric and depth channels are, respectively, captured by stacking several convolutional layers and a long short-term memory layer; the memory layer encodes both short-range and long-range spatial dependencies in an image along the vertical direction. Another long short-term memorized fusion layer is set up to integrate the contexts along the vertical direction from different channels, and perform bi-directional propagation of the fused vertical contexts along the horizontal direction to obtain true 2D global contexts. At last, the fused contextual representation is concatenated with the convolutional features extracted from the photometric channels in order to improve the accuracy of fine-scale semantic labeling. Our proposed model has set a new state of the art, i.e., ( 48.1 ) and ( 49.4 ) average class accuracy over 37 categories ( ( 2.2 ) and ( 5.4 ) improvement) on the large-scale SUNRGBD dataset and the NYUDv2 dataset, respectively. RGB-D images captured by consumer camera can provide pair-wise color and depth information but depth image usually contains strong noise and large holes. Due to different modalities of RGB-D, the intensity-guided depth enhancement easily causes artifacts and blurred edges. To solve this problem, we propose the multi-scale mutual feature convolutional neural network (MSMF-CNN) to learn essential mutual features of RGB-D. This end-to-end framework has two stages, i.e., mutual feature learning and depth regeneration. Firstly, we design two parallel separated subnetworks as intensity and depth mutual feature generator. Specifically, we design a multi-scale mutual feature generator to enforce the depth mutual feature learning and incorporate various structure characteristic. Secondly, depth transformation sub-network combines two mutual features to recover clean depth image. We tested our model on two well-known datasets in terms of PSNR, visual effect and computing speed. Compared with state-of-art algorithms, our method provides better performance of all three criteria.