Representation Learning: A Review and New Perspectives <s> Regularized Autoencoders Capture Local Structure of the Density <s> We present a series of theoretical arguments supporting the claim that a large class of modern learning algorithms that rely solely on the smoothness prior - with similarity between examples expressed with a local kernel - are sensitive to the curse of dimensionality, or more precisely to the variability of the target. Our discussion covers supervised, semi-supervised and unsupervised learning algorithms. These algorithms are found to be local in the sense that crucial properties of the learned function at x depend mostly on the neighbors of x in the training set. This makes them sensitive to the curse of dimensionality, well studied for classical non-parametric statistical learning. We show in the case of the Gaussian kernel that when the function to be learned has many variations, these algorithms require a number of training examples proportional to the number of variations, which could be large even though there may exist short descriptions of the target function, i.e. their Kolmogorov complexity may be low. This suggests that there exist non-local learning algorithms that at least have the potential to learn about such structured but apparently complex functions (because locally they have many variations), while not using very specific prior domain knowledge. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Regularized Autoencoders Capture Local Structure of the Density <s> To escape from the curse of dimensionality, we claim that one can learn non-local functions, in the sense that the value and shape of the learned function at x must be inferred using examples that may be far from x. With this objective, we present a non-local non-parametric density estimator. It builds upon previously proposed Gaussian mixture models with regularized covariance matrices to take into account the local shape of the manifold. It also builds upon recent work on non-local estimators of the tangent plane of a manifold, which are able to generalize in places with little training data, unlike traditional, local, non-parametric models. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Regularized Autoencoders Capture Local Structure of the Density <s> Denoising autoencoders have been previously shown to be competitive alternatives to restricted Boltzmann machines for unsupervised pretraining of each layer of a deep architecture. We show that a simple denoising autoencoder training criterion is equivalent to matching the score (with respect to the data) of a specific energy-based model to that of a nonparametric Parzen density estimator of the data. This yields several useful insights. It defines a proper probabilistic model for the denoising autoencoder technique, which makes it in principle possible to sample from them or rank examples by their energy. It suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives. It justifies the use of tied weights between the encoder and decoder and suggests ways to extend the success of denoising autoencoders to a larger family of energy-based models. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Regularized Autoencoders Capture Local Structure of the Density <s> We present in this paper a novel approach for training deterministic auto-encoders. We show that by adding a well chosen penalty term to the classical reconstruction cost function, we can achieve results that equal or surpass those attained by other regularized auto-encoders as well as denoising auto-encoders on a range of datasets. This penalty term corresponds to the Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input. We show that this penalty term results in a localized space contraction which in turn yields robust features on the activation layer. Furthermore, we show how this penalty term is related to both regularized auto-encoders and denoising auto-encoders and how it can be seen as a link between deterministic and non-deterministic auto-encoders. We find empirically that this penalty helps to carve a representation that better captures the local directions of variation dictated by the data, corresponding to a lower-dimensional non-linear manifold, while being more invariant to the vast majority of directions orthogonal to the manifold. Finally, we show that by using the learned features to initialize a MLP, we achieve state of the art classification error on a range of datasets, surpassing other methods of pretraining. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> Regularized Autoencoders Capture Local Structure of the Density <s> What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> Regularized Autoencoders Capture Local Structure of the Density <s> The contractive auto-encoder learns a representation of the input data that captures the local manifold structure around each data point, through the leading singular vectors of the Jacobian of the transformation from input to representation. The corresponding singular values specify how much local variation is plausible in directions associated with the corresponding singular vectors, while remaining in a high-density region of the input space. This paper proposes a procedure for generating samples that are consistent with the local structure captured by a contractive auto-encoder. The associated stochastic process defines a distribution from which one can sample, and which experimentally appears to converge quickly and mix well between modes, compared to Restricted Boltzmann Machines and Deep Belief Networks. The intuitions behind this procedure can also be used to train the second layer of contraction that pools lower-level features and learns to be invariant to the local directions of variation discovered in the first layer. We show that this can help learn and represent invariances present in the data and improve classification error. <s> BIB006 </s> Representation Learning: A Review and New Perspectives <s> Regularized Autoencoders Capture Local Structure of the Density <s> Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of the unknown data generating density. This paper contributes to the mathematical understanding of this phenomenon and helps define better justified sampling algorithms for deep learning based on auto-encoder variants. We consider an MCMC where each step samples from a Gaussian whose mean and covariance matrix depend on the previous state, defines through its asymptotic distribution a target density. First, we show that good choices (in the sense of consistency) for these mean and covariance functions are the local expected value and local covariance under that target density. Then we show that an auto-encoder with a contractive penalty captures estimators of these local moments in its reconstruction function and its Jacobian. A contribution of this work is thus a novel alternative to maximum-likelihood density estimation, which we call local moment matching. It also justifies a recently proposed sampling algorithm for the Contractive Auto-Encoder and extends it to the Denoising Auto-Encoder. <s> BIB007
Can we also say something about the probabilistic interpretation of regularized autoencoders? Their training criterion does not fit the standard likelihood framework because this would involve a data-dependent "prior." An interesting hypothesis emerges to answer that question, out of recent theoretical results BIB003 , BIB005 : The training criterion of regularized autoencoders, instead of being a form of maximum likelihood, corresponds to a different inductive principle, such as score matching. The score matching connection is discussed in Section 7.2.2 and has been shown for a particular parametrization of DAE and equivalent GRBM BIB003 . The work in BIB005 generalizes this idea to a broader class of parameterizations (arbitrary encoders and decoders) and shows that by regularizing the autoencoder so that it be contractive, one obtains that the reconstruction function and its derivative estimate first and second derivatives of the underlying data-generative density. This view can be exploited to successfully sample from autoencoders, as shown in BIB006 , BIB007 . The proposed sampling algorithms are MCMCs similar to Langevin MCMC, using not just the estimated first derivative of the density, but also the estimated manifold tangents so as to stay close to manifolds of high density. This interpretation connects well with the geometric perspective introduced in Section 8. The regularization effects (e.g., due to a sparsity regularizer, a contractive regularizer, or the denoising criterion) ask the learned representation to be as insensitive as possible to the input, while minimizing reconstruction error on the training examples forces the representation to contain just enough BIB001 . It yielded 0.81 percent error rate using the full MNIST training set, with no prior deformations, and no convolution. BIB002 . Suggested by Ian Goodfellow, personal communication. Fig. 4 . The tangent vectors to the high-density manifold as estimated by a contractive autoencoder BIB004 . The original input is shown on the top left. Each tangent vector (images on right side of first row) corresponds to a plausible additive deformation of the original input, as illustrated on the second row, where a bit of the third singular vector is added to the original, to form a translated and deformed image. Unlike in PCA, the tangent vectors are different for different inputs because the estimated manifold is highly nonlinear. information to distinguish them. The solution is that variations along the high-density manifolds are preserved, while other variations are compressed: The reconstruction function should be as constant as possible while reproducing training examples, i.e., points near a training example should be mapped to that training example (Fig. 5 ). The reconstruction function should map an input toward the nearest point manifold, i.e., the difference between reconstruction and input is a vector aligned with the estimated score (the derivative of the log density with respect to the input). The score can be zero on the manifold (where reconstruction error is also zero) at local maxima of the log density, but it can also be zero at local minima. It means that we cannot equate low reconstruction error with high estimated probability. The second derivatives of the log density correspond to the first derivatives of the reconstruction function, and on the manifold (where the first derivative is 0) they indicate the tangent directions of the manifold (where the first derivative remains near 0). As illustrated in Fig. 6 , the basic idea of the autoencoder sampling algorithms in BIB006 , BIB007 is to make MCMC moves where one 1) moves toward the manifold by following the density gradient (i.e., applying a reconstruction) and 2) adds noise in the directions of the leading singular vectors of the reconstruction (or encoder) Jacobian, corresponding to those associated with smallest second derivative of the log density.