Representation Learning: A Review and New Perspectives <s> Autoencoders <s> An autoencoder network uses a set of recognition weights to convert an input vector into a code vector. It then uses a set of generative weights to convert the code vector into an approximate reconstruction of the input vector. We derive an objective function for training autoencoders based on the Minimum Description Length (MDL) principle. The aim is to minimize the information required to describe both the code vector and the reconstruction error. We show that this information is minimized by choosing code vectors stochastically according to a Boltzmann distribution, where the generative weights define the energy of each possible code vector given the input vector. Unfortunately, if the code vectors use distributed representations, it is exponentially expensive to compute this Boltzmann distribution because it involves all possible code vectors. We show that the recognition weights of an autoencoder can be used to compute an approximation to the Boltzmann distribution and that this approximation gives an upper bound on the description length. Even when this bound is poor, it can be used as a Lyapunov function for learning both the generative and the recognition weights. We demonstrate that this approach can be used to learn factorial codes. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Autoencoders <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Autoencoders <s> The multilayer perceptron, when working in auto-association mode, is sometimes considered as an interesting candidate to perform data compression or dimensionality reduction of the feature space in information processing applications. The present paper shows that, for auto-association, the nonlinearities of the hidden units are useless and that the optimal parameter values can be derived directly by purely linear techniques relying on singular value decomposition and low rank matrix approximation, similar in spirit to the well-known Karhunen-Loève transform. This approach appears thus as an efficient alternative to the general error back-propagation algorithm commonly used for training multilayer perceptrons. Moreover, it also gives a clear interpretation of the rôle of the different parameters. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Autoencoders <s> Complexity theory of circuits strongly suggests that deep architectures can be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multi-layer neural networks have many levels of non-linearities allowing them to compactly represent highly non-linear and highly-varying functions. However, until recently it was not clear how to train such deep networks, since gradient-based optimization starting from random initialization appears to often get stuck in poor solutions. Hinton et al. recently introduced a greedy layer-wise unsupervised learning algorithm for Deep Belief Networks (DBN), a generative model with many layers of hidden causal variables. In the context of the above optimization problem, we study this algorithm empirically and explore variants to better understand its success and extend it to cases where the inputs are continuous or where the structure of the input distribution is not revealing enough about the variable to be predicted in a supervised task. Our experiments also confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> Autoencoders <s> We study an expansion of the log likelihood in undirected graphical models such as the restricted Boltzmann machine (RBM), where each term in the expansion is associated with a sample in a Gibbs chain alternating between two random variables (the visible vector and the hidden vector in RBMs). We are particularly interested in estimators of the gradient of the log likelihood obtained through this expansion. We show that its residual term converges to zero, justifying the use of a truncation---running only a short Gibbs chain, which is the main idea behind the contrastive divergence (CD) estimator of the log-likelihood gradient. By truncating even more, we obtain a stochastic reconstruction error, related through a mean-field approximation to the reconstruction error often used to train autoassociators and stacked autoassociators. The derivation is not specific to the particular parametric forms used in RBMs and requires only convergence of the Gibbs chain. We present theoretical and empirical evidence linking the number of Gibbs steps k and the magnitude of the RBM parameters to the bias in the CD estimator. These experiments also suggest that the sign of the CD estimator is correct most of the time, even when the bias is large, so that CD-k is a good descent direction even for small k. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> Autoencoders <s> Independent Components Analysis (ICA) and its variants have been successfully used for unsupervised feature learning. However, standard ICA requires an orthonoramlity constraint to be enforced, which makes it difficult to learn overcomplete features. In addition, ICA is sensitive to whitening. These properties make it challenging to scale ICA to high dimensional data. In this paper, we propose a robust soft reconstruction cost for ICA that allows us to learn highly overcomplete sparse features even on unwhitened data. Our formulation reveals formal connections between ICA and sparse autoencoders, which have previously been observed only empirically. Our algorithm can be used in conjunction with off-the-shelf fast unconstrained optimizers. We show that the soft reconstruction cost can also be used to prevent replicated features in tiled convolutional neural networks. Using our method to learn highly overcomplete sparse features and tiled convolutional neural networks, we obtain competitive performances on a wide variety of object recognition tasks. We achieve state-of-the-art test accuracies on the STL-10 and Hollywood2 datasets. <s> BIB006
In the autoencoder framework  , BIB003 , BIB001 , one starts by explicitly defining a feature-extracting function in a specific parameterized closed form. This function, which we will denote f , is called the encoder and will allow the straightforward and efficient computation of a feature vector h ¼ f ðxÞ from an input x. For each example x ðtÞ from a dataset fx ð1Þ ; . . . ; x ðT Þ g, we define where h ðtÞ is the feature vector or representation or code computed from x ðtÞ . Another closed-form parameterized function g , called the decoder, maps from feature space back into input space, producing a reconstruction r ¼ g ðhÞ. Whereas probabilistic models are defined from an explicit probability function and are trained to maximize (often approximately) the data likelihood (or a proxy), autoencoders are parameterized through their encoder and decoder and are trained using a different training principle. The set of parameters of the encoder and decoder are learned simultaneously on the task of reconstructing as well as possibly the original input, i.e., attempting to incur the lowest possible reconstruction error Lðx; rÞ-a measure of the discrepancy between x and its reconstruction r-over training examples. Good generalization means low reconstruction error at test examples, while having high reconstruction error for most other x configurations. To capture the structure of the data-generating distribution, it is therefore important that something in the training criterion or the parameterization prevents the autoencoder from learning the identity function, which has zero reconstruction error everywhere. This is achieved through various means in the different forms of autoencoders, as described below in more detail, and we call these regularized autoencoders. A particular form of regularization consists of constraining the code to have a low dimension, and this is what the classical autoencoder or PCA do. In summary, basic autoencoder training consists in finding a value of parameter vector minimizing reconstruction error: where x ðtÞ is a training example. This minimization is usually carried out by stochastic gradient descent as in the training of MLPs. Since autoencoders were primarily developed as MLPs, predicting their input, the most commonly used forms for the encoder and decoder are affine mappings, optionally followed by a nonlinearity: where s f and s g are the encoder and decoder activation functions (typically, the elementwise sigmoid or hyperbolic tangent nonlinearity, or the identity function if staying linear). The set of parameters of such a model is ¼ fW ; b; W 0 ; dg, where b and d are called encoder and decoder bias vectors and W and W 0 are the encoder and decoder weight matrices. The choice of s g and L depends largely on the input domain range and nature and is usually chosen so that L returns a negative log likelihood for the observed value of x. A natural choice for an unbounded domain is a linear decoder with a squared reconstruction error, i.e., s g ðaÞ ¼ a and Lðx; rÞ ¼ kx À rk 2 . If inputs are bounded between 0 and 1, however, ensuring a similarly bounded reconstruction can be achieved by using s g ¼ sigmoid. In addition, if the inputs are of a binary nature, a binary cross-entropy loss 14 is sometimes used. If both encoder and decoder use a sigmoid nonlinearity, then f ðxÞ and g ðhÞ have the exact same form as the conditionals P ðh j vÞ and P ðv j hÞ of binary RBMs (see Section 6.2). This similarity motivated an initial study BIB004 of the possibility of replacing RBMs with autoencoders as the basic pretraining strategy for building deep networks, as well as the comparative analysis of autoencoder reconstruction error gradient and CD updates BIB005 . One notable difference in the parameterization is that RBMs use a single weight matrix, which follows naturally from their energy function, whereas the autoencoder framework allows for a different matrix in the encoder and decoder. In practice, however, weight tying in which one defines W 0 ¼ W T may be (and is most often) used, rendering the parameterizations identical. The usual training procedures, however, differ greatly between the two approaches. A practical advantage of training autoencoder variants is that they define a simple tractable optimization objective that can be used to monitor progress. In the case of a linear autoencoder (linear encoder and decoder) with squared reconstruction error, minimizing BIB002 learns the same subspace 15 as PCA. This is also true when using a sigmoid nonlinearity in the encoder BIB003 , but not if the weights W and W 0 are tied (W 0 ¼ W T ) because W cannot be forced into being small and W 0 large to achieve a linear encoder. Similarly, Le et al. BIB006 recently showed that adding a regularization term of the form P t P j s 3 ðW j x ðtÞ Þ to a linear autoencoder with tied weights, where s 3 is a nonlinear convex function, yields an efficient algorithm for learning linear ICA.