A Survey on Deep Learning Methods for Robot Vision <s> Fully Convolutional Networks <s> High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such “autoencoder” networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data. <s> BIB001 </s> A Survey on Deep Learning Methods for Robot Vision <s> Fully Convolutional Networks <s> We explore an original strategy for building deep networks, based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs. The resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders. It is however shown on a benchmark of classification problems to yield significantly lower classification error, thus bridging the performance gap with deep belief networks (DBN), and in several cases surpassing it. Higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent SVM classifiers. Qualitative experiments show that, contrary to ordinary autoencoders, denoising autoencoders are able to learn Gabor-like edge detectors from natural image patches and larger stroke detectors from digit images. This work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations. <s> BIB002 </s> A Survey on Deep Learning Methods for Robot Vision <s> Fully Convolutional Networks <s> Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet [20], the VGG net [31], and GoogLeNet [32]) into fully convolutional networks and transfer their learned representations by fine-tuning [3] to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes less than one fifth of a second for a typical image. <s> BIB003 </s> A Survey on Deep Learning Methods for Robot Vision <s> Fully Convolutional Networks <s> We present a deep learning framework for probabilistic pixel-wise semantic segmentation, which we term Bayesian SegNet. Semantic segmentation is an important tool for visual scene understanding and a meaningful measure of uncertainty is essential for decision making. Our contribution is a practical system which is able to predict pixel-wise class labels with a measure of model uncertainty. We achieve this by Monte Carlo sampling with dropout at test time to generate a posterior distribution of pixel class labels. In addition, we show that modelling uncertainty improves segmentation performance by 2-3% across a number of state of the art architectures such as SegNet, FCN and Dilation Network, with no additional parametrisation. We also observe a significant improvement in performance for smaller datasets where modelling uncertainty is more effective. We benchmark Bayesian SegNet on the indoor SUN Scene Understanding and outdoor CamVid driving scenes datasets. <s> BIB004 </s> A Survey on Deep Learning Methods for Robot Vision <s> Fully Convolutional Networks <s> State-of-the-art models for semantic segmentation are based on adaptations of convolutional networks that had originally been designed for image classification. However, dense prediction and image classification are structurally different. In this work, we develop a new convolutional network module that is specifically designed for dense prediction. The presented module uses dilated convolutions to systematically aggregate multi-scale contextual information without losing resolution. The architecture is based on the fact that dilated convolutions support exponential expansion of the receptive field without loss of resolution or coverage. We show that the presented context module increases the accuracy of state-of-the-art semantic segmentation systems. In addition, we examine the adaptation of image classification networks to dense prediction and show that simplifying the adapted network can increase accuracy. <s> BIB005 </s> A Survey on Deep Learning Methods for Robot Vision <s> Fully Convolutional Networks <s> Recent advances in semantic image segmentation have mostly been achieved by training deep convolutional neural networks (CNNs). We show how to improve semantic segmentation through the use of contextual information; specifically, we explore `patch-patch' context between image regions, and `patch-background' context. For learning from the patch-patch context, we formulate Conditional Random Fields (CRFs) with CNN-based pairwise potential functions to capture semantic correlations between neighboring patches. Efficient piecewise training of the proposed deep structured model is then applied to avoid repeated expensive CRF inference for back propagation. For capturing the patch-background context, we show that a network design with traditional multi-scale image input and sliding pyramid pooling is effective for improving performance. Our experimental results set new state-of-the-art performance on a number of popular semantic segmentation datasets, including NYUDv2, PASCAL VOC 2012, PASCAL-Context, and SIFT-flow. In particular, we achieve an intersection-over-union score of 78.0 on the challenging PASCAL VOC 2012 dataset. <s> BIB006 </s> A Survey on Deep Learning Methods for Robot Vision <s> Fully Convolutional Networks <s> We present a novel and practical deep fully convolutional neural network architecture for semantic pixel-wise segmentation termed SegNet. This core trainable segmentation engine consists of an encoder network, a corresponding decoder network followed by a pixel-wise classification layer. The architecture of the encoder network is topologically identical to the 13 convolutional layers in the VGG16 network [1] . The role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. The novelty of SegNet lies is in the manner in which the decoder upsamples its lower resolution input feature map(s). Specifically, the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. This eliminates the need for learning to upsample. The upsampled maps are sparse and are then convolved with trainable filters to produce dense feature maps. We compare our proposed architecture with the widely adopted FCN [2] and also with the well known DeepLab-LargeFOV [3] , DeconvNet [4] architectures. This comparison reveals the memory versus accuracy trade-off involved in achieving good segmentation performance. SegNet was primarily motivated by scene understanding applications. Hence, it is designed to be efficient both in terms of memory and computational time during inference. It is also significantly smaller in the number of trainable parameters than other competing architectures and can be trained end-to-end using stochastic gradient descent. We also performed a controlled benchmark of SegNet and other architectures on both road scenes and SUN RGB-D indoor scene segmentation tasks. These quantitative assessments show that SegNet provides good performance with competitive inference time and most efficient inference memory-wise as compared to other architectures. We also provide a Caffe implementation of SegNet and a web demo at http://mi.eng.cam.ac.uk/projects/segnet. <s> BIB007 </s> A Survey on Deep Learning Methods for Robot Vision <s> Fully Convolutional Networks <s> In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First , we highlight convolution with upsampled filters, or ‘atrous convolution’, as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second , we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third , we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed “DeepLab” system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online. <s> BIB008
Fully convolutional networks (FCNs) are a special kind of CNN that includes only convolutional layers (no fully connected units). The output of an FCN is a feature map from the last convolutional layer. Given that FCN inputs and outputs are two-dimensional, they are trained end-to-end, pixel-to-pixel, and can be used for dimensionality reduction, regression, or semantic segmentation. Auto-encoders BIB001 are artificial neural networks trained to predict their own inputs. FCN based auto-encoders are formed by two smaller FCNs: an encoder and a decoder. The encoder is an FCN that decreases the size of the input image through its layers, generating a compressed representation of the image. The decoder is an FCN that maps the compressed representation onto a final image. When the auto-encoder is trained, its encoder is able to compress images, while its decoder can reconstruct them. Denoising auto-encoders BIB002 are FCN regression models that are able to denoise images. They are trained with noisy images as input, and their correspondent clean images as output. Also, FCNs can be used for performing semantic segmentation BIB003 [31] BIB004 , sometimes in conjunction with Conditional Random Fields (CRFs) BIB005 . Semantic segmentation FCNs are trained to map RGB images onto images containing a pixel-wise semantic segmentation of the environment. The first successful use of FCNs for semantic segmentation was implemented in the work of Long, Shelhamer and Darrell BIB003 . They used a set of convolutional layers with decreasing size whose output corresponds to the segmented image with a lower resolution than the original one. They used dropout during the training stage for regularization. SegNet BIB007 is also based on an encoder and a decoder. The encoder corresponds to a set of convolutional layers with decreasing spatial size. The decoder takes the output data of the encoder as input, and applies a set of convolutional layers that scale up the data to become the same size as the original input image. By using the decoder, the segmentation obtained by SegNet has higher resolution than that of Long et al. BIB003 . The SegNet architecture is shown in Figure 3 . Bayesian SegNet BIB004 uses dropout both for training and for normal execution. Since dropout selects random subsets of the neurons when using the network, various network outputs can be obtained by applying dropout several times. Then, a set of semantic, segmented images is available. The images can be analyzed for computing the uncertainty associated with the labels of each pixel. Then, if a pixel has the same label in all of the images, the uncertainty is low, but if the pixel has different labels in the images, their uncertainty is high. FCNs are based on simple pixel-wise CNNs, without any explicit modeling of the image structure. Thus, current research on semantic segmentation is focused on modifying the baseline FCNs for using contextual information. Dilation10 BIB005 and DeepLabv2-CRF BIB008 use dilated/up-sampled filters for covering a greater receptive field without using more parameters. LLR-4x [84] uses a multi-scale representation similar to a Laplace pyramid for collecting semantically and spatially resolved data. DeepLabv2-CRF BIB008 and Adelaide-Context BIB006 also integrate CRFs into the FCN process. 