A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> 1. The striate cortex was studied in lightly anaesthetized macaque and spider monkeys by recording extracellularly from single units and stimulating the retinas with spots or patterns of light. Most cells can be categorized as simple, complex, or hypercomplex, with response properties very similar to those previously described in the cat. On the average, however, receptive fields are smaller, and there is a greater sensitivity to changes in stimulus orientation. A small proportion of the cells are colour coded.2. Evidence is presented for at least two independent systems of columns extending vertically from surface to white matter. Columns of the first type contain cells with common receptive-field orientations. They are similar to the orientation columns described in the cat, but are probably smaller in cross-sectional area. In the second system cells are aggregated into columns according to eye preference. The ocular dominance columns are larger than the orientation columns, and the two sets of boundaries seem to be independent.3. There is a tendency for cells to be grouped according to symmetry of responses to movement; in some regions the cells respond equally well to the two opposite directions of movement of a line, but other regions contain a mixture of cells favouring one direction and cells favouring the other.4. A horizontal organization corresponding to the cortical layering can also be discerned. The upper layers (II and the upper two-thirds of III) contain complex and hypercomplex cells, but simple cells are virtually absent. The cells are mostly binocularly driven. Simple cells are found deep in layer III, and in IV A and IV B. In layer IV B they form a large proportion of the population, whereas complex cells are rare. In layers IV A and IV B one finds units lacking orientation specificity; it is not clear whether these are cell bodies or axons of geniculate cells. In layer IV most cells are driven by one eye only; this layer consists of a mosaic with cells of some regions responding to one eye only, those of other regions responding to the other eye. Layers V and VI contain mostly complex and hypercomplex cells, binocularly driven.5. The cortex is seen as a system organized vertically and horizontally in entirely different ways. In the vertical system (in which cells lying along a vertical line in the cortex have common features) stimulus dimensions such as retinal position, line orientation, ocular dominance, and perhaps directionality of movement, are mapped in sets of superimposed but independent mosaics. The horizontal system segregates cells in layers by hierarchical orders, the lowest orders (simple cells monocularly driven) located in and near layer IV, the higher orders in the upper and lower layers. <s> BIB001 </s> A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> This paper presents a novel pedestrian detection method based on the use of a convolutional neural network (CNN) classifier. Our method achieves high accuracy by automatically optimizing the feature representation to the detection task and regularizing the neural network. We evaluate the proposed method on a difficult database containing pedestrians in a city environment with no restrictions on pose, action, background and lighting conditions. The false positive rate (FPR) of the proposed CNN classifier is less than 1/5-th of the FPR of a support vector machine (SVM) classifier using Haar-wavelet features when the detection rate is 90%. The accuracy of the SVM classifier using the features learnt by the CNN is equivalent to the accuracy of the CNN, confirming the importance of automatically optimized features. The computational demand of the CNN classifier is, however, more than an order of magnitude lower than that of the SVM, irrespective of the type of features used. <s> BIB002 </s> A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> For object recognition, the current state-of-the-art is based on exhaustive search. However, to enable the use of more expensive features and classifiers and thereby progress beyond the state-of-the-art, a selective search strategy is needed. Therefore, we adapt segmentation as a selective search by reconsidering segmentation: We propose to generate many approximate locations over few and precise object delineations because (1) an object whose location is never generated can not be recognised and (2) appearance and immediate nearby context are most effective for object recognition. Our method is class-independent and is shown to cover 96.7% of all objects in the Pascal VOC 2007 test set using only 1,536 locations per image. Our selective search enables the use of the more expensive bag-of-words method which we use to substantially improve the state-of-the-art by up to 8.5% for 8 out of 20 classes on the Pascal VOC 2010 detection challenge. <s> BIB003 </s> A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> Traditional methods of computer vision and machine learning cannot match human performance on tasks such as the recognition of handwritten digits or traffic signs. Our biologically plausible, wide and deep artificial neural network architectures can. Small (often minimal) receptive fields of convolutional winner-take-all neurons yield large network depth, resulting in roughly as many sparsely connected neural layers as found in mammals between retina and visual cortex. Only winner neurons are trained. Several deep neural columns become experts on inputs preprocessed in different ways; their predictions are averaged. Graphics cards allow for fast training. On the very competitive MNIST handwriting benchmark, our method is the first to achieve near-human performance. On a traffic sign recognition benchmark it outperforms humans by a factor of two. We also improve the state-of-the-art on a plethora of common image classification benchmarks. <s> BIB004 </s> A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision. <s> BIB005 </s> A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> Object detection performance, as measured on the canonical PASCAL VOC dataset, has plateaued in the last few years. The best-performing methods are complex ensemble systems that typically combine multiple low-level image features with high-level context. In this paper, we propose a simple and scalable detection algorithm that improves mean average precision (mAP) by more than 30% relative to the previous best result on VOC 2012 -- achieving a mAP of 53.3%. Our approach combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost. Since we combine region proposals with CNNs, we call our method R-CNN: Regions with CNN features. We also present experiments that provide insight into what the network learns, revealing a rich hierarchy of image features. Source code for the complete system is available at http://www.cs.berkeley.edu/~rbg/rcnn. <s> BIB006 </s> A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors. <s> BIB007 </s> A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. <s> BIB008 </s> A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62 % error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers. <s> BIB009 </s> A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0%, respectively, which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully connected layers we employed a recently developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB010 </s> A Survey on Deep Learning Methods for Robot Vision <s> Convolutional Neural Networks <s> State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available. <s> BIB011
Convolutional Neural Networks (CNNs) correspond to a family of DNN architectures whose main goal is analyzing natural signals by taking advantage of their properties. According to BIB007 , the main properties of CNNs are local connections, shared weights, pooling, and the use of many layers. The main reason for this is that the structure of CNNs is inspired by the processing pipeline of the mammalian visual system BIB001 , where (i) neurons have local connectivity, (ii) several layers of neurons exist between the fovea and the cortical visual areas, (iii) neurons on a given layer perform the same kind of processing (a non-linear activation function is applied over its weighted inputs), and (iv) the size of the receptive fields of the neurons (i.e. the region of the sensory space whose signals affect the operation of a given neuron) increases from one layer to the next one. Hence, the behavior of a given layer of CNN neurons can be modeled as a linear filtering operation, i.e. a convolution, followed by a non-linear operation (activation function). After the convolution layer, the application of a pooling operation permits modeling the increase of the receptive size of the neurons. After several convolutions and pooling operations are applied (the network has several layers), a decision needs to be made about the input stimuli. This is implemented using fully connected layers of neurons. Thus, CNNs are composed of three kinds of processing layers: convolutional layers, pooling layers, and fully-connected layers, as is shown in Figure 1 . The number of layers in a CNN and their type will depend on the specific network structure, but normally the number of convolution and pooling layers is much larger than the number of fully-connected layers. A very important side effect of this is that a CNN can have a smaller number of weights to be adapted than a shallow network, given that the number of free parameters in the convolutional layers is very small, and proportional to the size of the implemented filters. For instance, in a CNN whose first convolutional layer has 224x224x3 input neurons (i.e., 3 channels), a filter size of 11x11, and C output channels, the number of free parameters, i.e. the weights, to be adapted is 11x11x3xC, while in the case of a fully-connected network, the number of weights would be 224*224*3*C. Then, the number of free parameters increases linearly if the input has several channels (i.e., a RGB input image) and/or if several output channels must be computed. One important aspect to highlight is that most of the operations that need to be computed in a CNN are local, and therefore can be mapped easily to GPUs. The designing of a convolutional layer basically consists of selecting the number of filters that a given layer will have, and its size. The parameters of the filters, i.e. the weights of the neurons, will be adapted during training. In the case of the design of a pooling layer, the pooling operation needs to be selected. The most frequently used pooling operations are average pooling and max pooling; however, several other alternatives have been studied and used  (e.g. stochastic pooling, spatial pyramid pooling, def-pooling). The fully-connected layers are the ones in charge of converting the 2D feature maps into a 1D feature vector, in addition to being in charge of the classification. The fully-connected layers have a similar structure to a traditional shallow network, and they normally contain a large percentage of the parameters of a CNN (about 90% according to  ). In some cases the fully-connected layers in charge of the final classification are replaced by a softmax unit, or a different statistical classifier (e.g. an SVM BIB002 ). Softmax has been shown to achieve better accuracy than SVM  , because it can be trained end-to-end inside the network. Adapted from  . Different CNN architectures implement different network topologies (e.g. number of convolutional layers, filter size in each layer, number of fully connected layers), and operations (e.g. activation function, pooling operation, dropout). AlexNet, one of the most popular CNNs and the one responsible for the DNN boom thanks to its outstanding performance in ILSVRC2012, has 5 convolutional layers and 3 fully connected layers. The convolutional layers 1, 2, and 5 include max pooling, and the convolutional layers 1 and 2 include local response normalization BIB010 . In addition, they use ReLU activation functions, dropout, and stochastic gradient descent for training. AlexNet has a fixed input size (224x224x3) and is well suited for image classification tasks. The network generates a probability vector of size 1000 from the whole input image, each component representing the probability of presence for one object category. Some CNN architectures keep the same working strategy as AlexNet, but they go deeper. For instance, VGG BIB005 , which got the second place at ILSVRC 2014, has 16 (18) layers, 13 (15) convolutional and 3 fully-connected, but it reduces the filter size to 3x3. GoogleNet  , which won first place at ILSVRC 2014, has 22 layers, and introduced a more complex architecture based on the use of several parallel sub-layers inside each layer. However, in practice, the use of 30 or more layers generates an increase in the error. This problem is solved by ResNet BIB008 by using residual layers that work by adding the output of a previous layer to the current one. Then, the network processes residuals instead of the original data layers. ResNet is able to use up to 1001 layers BIB009 , consistently decreasing the error when adding more layers. While ResNet uses only skip connections between pairs of layers, DenseNet  uses skip connections between every layer and its previous layers. DenseNet has been shown to perform better than ResNet, while requiring fewer parameters to learn. Finally squeeze-and-excitation networks, (SENet) ,  were introduced On the other hand, some approaches focus on using multiple CNNs that work independently, and whose results are combined. One of the most interesting approaches is the Multi-Column DNN (MCDNN) BIB004 , which uses several DNNs in parallel and averages their predictions. MCDNN-based approaches obtain state-of-the-art results on some image classification benchmarks (see Table 2 ). detection/recognition tasks, where the object of interest covers a small area in the input image, the CNN will not be able to detect the object. In that case, sliding windows can be used for defining the areas of the image to be analyzed. However, this approach, which requires analyzing all possible sliding windows using a CNN, is unable to run in near real-time because of the large number of windows to be processed. Hence, instead of using a sliding window before applying CNNs, region proposals can be computed. A region proposal is a window inside the input image that is likely to contain some object. Region proposal algorithms are classagnostic object detectors that can be computed very quickly. The R-CNN algorithm BIB006 is based on using region proposals that are fed into a CNN. The proposals are computed by using the Selective Search algorithm BIB003 . The R-CNN algorithm is able to run faster than a naïve CNN using sliding windows, but the CNN features need to be recomputed for each object proposal instance. Fast R-CNN  also employs object proposals from Selective Search, but the features to be used for the final detection are computed just once, using shared convolutional layers for each input image. Each proposal defines a region of interest in the last feature map, and the selected region is used for carrying out the classification, using fully connected layers. Faster R-CNN BIB011 implements a Region Proposal Network (RPN) that is computed by applying a per-pixel fully connected network over the last shared convolutional layer (see Figure 2 ). The RPN network is able to generate region proposals, which are then used by a Fast R-CNN for the detection of the objects. The Faster R-CNN has also been used with the ResNet architecture BIB008 [24], and it is the state-of-the-art for general-purpose object detection (see Table 2 ). 