A Survey on Deep Learning Methods for Robot Vision <s> Basic Definitions <s> Abstract We show that standard multilayer feedforward networks with as few as a single hidden layer and arbitrary bounded and nonconstant activation function are universal approximators with respect to L p (μ) performance criteria, for arbitrary finite input environment measures μ, provided only that sufficiently many hidden units are available. If the activation function is continuous, bounded and nonconstant, then continuous mappings can be learned uniformly over compact input sets. We also give very general conditions ensuring that networks with sufficiently smooth activation functions are capable of arbitrarily accurate approximation to a function and its derivatives. <s> BIB001 </s> A Survey on Deep Learning Methods for Robot Vision <s> Basic Definitions <s> During the last decade, the data sizes have grown faster than the speed of processors. In this context, the capabilities of statistical machine learning methods is limited by the computing time rather than the sample size. A more precise analysis uncovers qualitatively different tradeoffs for the case of small-scale and large-scale learning problems. The large-scale case involves the computational complexity of the underlying optimization algorithm in non-trivial ways. Unlikely optimization algorithms such as stochastic gradient descent show amazing performance for large-scale problems. In particular, second order stochastic gradient and averaged stochastic gradient are asymptotically efficient after a single pass on the training set. <s> BIB002 </s> A Survey on Deep Learning Methods for Robot Vision <s> Basic Definitions <s> When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition. <s> BIB003 </s> A Survey on Deep Learning Methods for Robot Vision <s> Basic Definitions <s> We introduce DropConnect, a generalization of Dropout (Hinton et al., 2012), for regularizing large fully-connected layers within neural networks. When training with Dropout, a randomly selected subset of activations are set to zero within each layer. DropConnect instead sets a randomly selected subset of weights within the network to zero. Each unit thus receives input from a random subset of units in the previous layer. We derive a bound on the generalization performance of both Dropout and DropConnect. We then evaluate DropConnect on a range of datasets, comparing to Dropout, and show state-of-the-art results on several image recognition benchmarks by aggregating multiple DropConnect-trained models. <s> BIB004 </s> A Survey on Deep Learning Methods for Robot Vision <s> Basic Definitions <s> A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum. Here we argue, based on results from statistical physics, random matrix theory, neural network theory, and empirical evidence, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest. Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum. Motivated by these arguments, we propose a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods. We apply this algorithm to deep or recurrent neural network training, and provide numerical evidence for its superior optimization performance. <s> BIB005 </s> A Survey on Deep Learning Methods for Robot Vision <s> Basic Definitions <s> Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors. <s> BIB006 </s> A Survey on Deep Learning Methods for Robot Vision <s> Basic Definitions <s> We present a deep learning framework for probabilistic pixel-wise semantic segmentation, which we term Bayesian SegNet. Semantic segmentation is an important tool for visual scene understanding and a meaningful measure of uncertainty is essential for decision making. Our contribution is a practical system which is able to predict pixel-wise class labels with a measure of model uncertainty. We achieve this by Monte Carlo sampling with dropout at test time to generate a posterior distribution of pixel class labels. In addition, we show that modelling uncertainty improves segmentation performance by 2-3% across a number of state of the art architectures such as SegNet, FCN and Dilation Network, with no additional parametrisation. We also observe a significant improvement in performance for smaller datasets where modelling uncertainty is more effective. We benchmark Bayesian SegNet on the indoor SUN Scene Understanding and outdoor CamVid driving scenes datasets. <s> BIB007 </s> A Survey on Deep Learning Methods for Robot Vision <s> Basic Definitions <s> Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters. <s> BIB008 </s> A Survey on Deep Learning Methods for Robot Vision <s> Basic Definitions <s> Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. <s> BIB009
An artificial neuron takes on an input vector and outputs a scalar value. The neuron is parameterized by a set of weights. Each weight is used as a multiplier for a scalar input. The output of the neuron is the result of applying a nonlinear activation function on the sum of the weighted inputs. Thus, a neuron with weights w , inputs x , output y , and non-linear activation function φ is represented as: with n the size of x , and 0 w representing a bias for the neuron input if 0 x is set to 1. Non-linear activation functions ) (x φ are used for improving the expressive power of the network. They are valid as long as they are continuous, bounded, and monotonically increasing BIB001 . An additional requirement of the learning algorithms over ) (x φ is differentiability. Thus, the activation functions used most often include the sigmoid logistic function φ and to the parameters of the learning process. The parameters of a network correspond to the set of weights of all the neurons. The network can be trained for selecting optimal weights (parameters) by using numerical optimization methods. The optimal hyperparameters cannot be learned directly from the data, so several network architectures must be trained and tested for selecting the best hyper-parameters. Thus, in order to be able to adapt the parameters of the network, to select its hyper-parameters, and to characterize its performance, the available data must be divided into three non-overlapping datasets: training, validation, and test. The training dataset is used for learning the model parameters (weights) of the network. The validation dataset is used for selecting the hyper-parameters by evaluating the performance of the network under different hyper-parameter configurations. The test dataset is used for characterizing the trained network by estimating its generalization error. DNNs are usually trained using supervised learning. The objective of the training procedure is to find the network parameters that minimize a loss function (e.g. a function of the global error between the obtained and the desired output of the network). Two kinds of iterative optimization procedures are commonly used: first order optimization (gradient based), and second order optimization (hessian based). When the loss function is convex, second order optimization is commonly used because it converges faster to the optimal solution. However, in non-convex problems, second order algorithms can be attracted to saddle points or local minima BIB005  . Then, gradient-based methods are normally used for training deep neural networks. As the datasets used for training deep neural networks are generally large, only a small subset of the data (a mini-batch) is used at each optimization iteration for training the network. This procedure is called Stochastic Gradient Descent (SGD) BIB002 . The term stochastic is used because each mini-batch gives a noisy estimation of the gradient over all the data, which allows finding a good set of weights very quickly BIB006 . "The stochastic gradient descent directly optimizes the expected risk, since the examples are randomly drawn from the ground truth distribution." BIB002 . The use of mini-batches improves the generalization ability of the network and enables fast training. Additional mechanisms are the decrease of the learning rate over time, and the use of momentum  . The generation of specialized second order algorithms for deep learning is a current research topic BIB005 . In several cases, minimizing the original loss function can cause misbehavior of the optimization process. For example, in some cases the learned weights can be too large because of overfitting. When the training procedure generates ill-posed networks, regularization can be used. Regularization corresponds to a set of techniques used to reduce overfitting by improving robustness in the parameter estimation process  . Some of these techniques can be implemented by adding a regularization term to the loss function. This extra term can be used for limiting the size of the weights, and also for improving invariance against small changes in the inputs. As an example, L2 regularization  consists of adding a penalization term to the original loss function: Dropout BIB003 [63] is a technique that consists of using only a random subset of the neurons in each layer (along with its connections) during training episodes in order to reduce overfitting. This procedure is able to improve the generalization error of the networks, since its effect is similar to that of using an ensemble of networks. Using dropout can be interpreted as an alternative form of regularization. Once the network has been trained, weights are decreased proportionally in order to account for the smaller number of neurons used when training. Also, dropout can be used for generating multiple possible outputs for the same input BIB007 when activating different units. In this way, a confidence level for the output can be computed. An alternative to Dropout is DropConnect BIB004 , which, instead of dropping the activations of some neurons, drops their weights. In this way each unit receives input from a random subset of units on the previous layer. Batch normalization consists of normalizing the input layers at each mini-batch iteration BIB008 . This enables the training procedure to use higher learning rates, and to be more robust with respect to initial weights in the network. Batch normalization also acts as a regularizer, eliminating the need for dropout. State-of-the-art ResNet networks, described in Section 2.3, do not use dropout; they use batch normalization BIB009 .