Representation Learning: A Review and New Perspectives <s> SINGLE-LAYER LEARNING MODULES <s> Adaptive sparse coding methods learn a possibly overcomplete set of basis functions, such that natural image patches can be reconstructed by linearly combining a small subset of these bases. The applicability of these methods to visual object recognition tasks has been limited because of the prohibitive cost of the optimization algorithms required to compute the sparse representation. In this work we propose a simple and efficient algorithm to learn basis functions. After training, this model also provides a fast and smooth approximator to the optimal representation, achieving even better accuracy than exact sparse coding algorithms on visual object recognition tasks. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> SINGLE-LAYER LEARNING MODULES <s> Denoising autoencoders have been previously shown to be competitive alternatives to restricted Boltzmann machines for unsupervised pretraining of each layer of a deep architecture. We show that a simple denoising autoencoder training criterion is equivalent to matching the score (with respect to the data) of a specific energy-based model to that of a nonparametric Parzen density estimator of the data. This yields several useful insights. It defines a proper probabilistic model for the denoising autoencoder technique, which makes it in principle possible to sample from them or rank examples by their energy. It suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives. It justifies the use of tied weights between the encoder and decoder and suggests ways to extend the success of denoising autoencoders to a larger family of energy-based models. <s> BIB002
Within the community of researchers interested in representation learning, there has developed two broad parallel lines of inquiry: one rooted in probabilistic graphical models and one rooted in neural networks. Fundamentally, the difference between these two paradigms is whether the layered architecture of a deep learning model is to be interpreted as describing a probabilistic graphical model or as describing a computation graph. In short, are hidden units considered latent random variables or as computational nodes? To date, the dichotomy between these two paradigms has remained in the background, perhaps because they appear to have more characteristics in common than separating them. We suggest that this is likely a function of the fact that much recent progress in both of these areas has focused on single-layer greedy learning modules and the similarities between the types of single-layer models that have been explored: mainly, the RBM on the probabilistic side and the autoencoder variants on the neural network side. Indeed, as shown by one of us BIB002 and others  , in the case of the RBM, training the model via an inductive principle known as score matching  (to be discussed in Section 6.4.3) is essentially identical to applying a regularized reconstruction objective to an autoencoder. Another strong link between pairs of models on both sides of this divide is when the computational graph for computing representation in the neural network model corresponds exactly to the computational graph that corresponds to inference in the probabilistic model, and this happens to also correspond to the structure of graphical model itself (e.g., as in the RBM). The connection between these two paradigms becomes more tenuous when we consider deeper models, where in the case of a probabilistic model, exact inference typically becomes intractable. In the case of deep models, the computational graph diverges from the structure of the model. For example, in the case of a DBM, unrolling variational (approximate) inference into a computational graph results in a recurrent graph structure. We have performed preliminary exploration  of deterministic variants of deep autoencoders whose computational graph is similar to that of a DBM (in fact very close to the meanfield variational approximations associated with the Boltzmann machine), and that is one interesting intermediate point to explore (between the deterministic approaches and the graphical model approaches). In the next few sections, we will review the major developments in single-layer training modules used to support feature learning and particularly deep learning. We divide these sections between (Section 6) the probabilistic models, with inference and training schemes that directly parameterize the generative-or decoding-pathway and (Section 7) the typically neural network-based models that directly parametrize the encoding pathway. Interestingly, some models like predictive sparse decomposition (PSD) BIB001 inherit both properties and will also be discussed (Section 7.2.4). We then present a different view of representation learning, based on the associated geometry and the manifold assumption, in Section 8. First, let us consider an unsupervised single-layer representation learning algorithm spanning all three views: probabilistic, autoencoder, and manifold learning.