A Survey on Deep Learning Methods for Robot Vision <s> Restricted Boltzmann Machines <s> Deep generative models with multiple hidden layers have been shown to be able to learn meaningful and compact representations of data. In this work we propose deep energy models, which use deep feedforward neural networks to model the energy landscapes that define probabilistic models. We are able to efficiently train all layers of our model simultaneously, allowing the lower layers of the model to adapt to the training of the higher layers, and thereby producing better generative models. We evaluate the generative performance of our models on natural images and demonstrate that this joint training of multiple layers yields qualitative and quantitative improvements over greedy layerwise training. We further generalize our models beyond the commonly used sigmoidal neural networks and show how a deep extension of the product of Student-t distributions model achieves good generative performance. Finally, we introduce a discriminative extension of our model and demonstrate that it outperforms other fully-connected models on object recognition on the NORB dataset. <s> BIB001 </s> A Survey on Deep Learning Methods for Robot Vision <s> Restricted Boltzmann Machines <s> Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. <s> BIB002
Restricted Boltzmann Machines (RBMs) are stochastic neural networks composed of stochastic neuron-like binary units. RBMs consist of one layer of visible units, one layer of hidden units, and an extra bias unit. Each visible unit is connected to all hidden units and vice versa. The bias unit is also connected to all hidden and visible units. All of the connections are undirected. RBMs are able to learn the data distribution of the input data (visible units) and to generate samples from that distribution (generative learning) in an unsupervised fashion. The network is able to approximate any distribution as the number of hidden units increases. Given an initial visible input, the network is able to compute binary activations for the hidden units by using binary sampling. Also, given the values of the hidden units, the network can compute activation values for the visible units. By repeating these two processes, the underlying distribution represented by the RBM can be sampled. The parameters of the network are obtained by minimizing an energy function that depends on the training input vectors (no supervision is required). In practice, parameters are adapted for diminishing the divergence between the distribution sampled from the RBM and the ground truth distribution. Also, several layers of hidden units can be stacked for creating a Deep Boltzmann Machine (DBM), but its training procedure requires high computational resources. Another form of stacking RBMs is by using Deep Belief Networks (DBNs) that include directed connections between layers, enabling the use of greedy layer-wise training algorithms. A more recent approach for training unsupervised stochastic networks is the Deep Energy Model (DEM) BIB001 . This architecture is composed of a set of deterministic feed forward layers and a final stochastic layer. This system can also be described as a deep feed-forward network followed by an RBM, trained jointly. Then, the feed-forward network learns a representation that is easily modeled by the final RBM. The system can also be interpreted as a normal RBM with a modified energy function that is dependent on the parameters of the feed-forward network. The training of DEMs is faster than the training of DBMs and DBNs, because only one stochastic layer is used. DBMs/DBNs/DEMs can be used as a pre-training step for initializing the weights of feed-forward networks. However, although state-of-the-art feed-forward implementations still do not use RBM-derived methods for pre-training BIB002 [80], this tendency could change in the future.