Representation Learning: A Review and New Perspectives <s> DAEs <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> DAEs <s> Previous work has shown that the difficulties in learning deep generative or discriminative models can be overcome by an initial unsupervised learning step that maps inputs to useful intermediate representations. We introduce and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern. This approach can be used to train autoencoders, and these denoising autoencoders can be stacked to initialize deep architectures. The algorithm can be motivated from a manifold learning and information theoretic perspective or from a generative model perspective. Comparative experiments clearly show the surprising advantage of corrupting the input of autoencoders on a pattern classification benchmark suite. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> DAEs <s> We explore an original strategy for building deep networks, based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs. The resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders. It is however shown on a benchmark of classification problems to yield significantly lower classification error, thus bridging the performance gap with deep belief networks (DBN), and in several cases surpassing it. Higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent SVM classifiers. Qualitative experiments show that, contrary to ordinary autoencoders, denoising autoencoders are able to learn Gabor-like edge detectors from natural image patches and larger stroke detectors from digit images. This work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> DAEs <s> Denoising autoencoders have been previously shown to be competitive alternatives to restricted Boltzmann machines for unsupervised pretraining of each layer of a deep architecture. We show that a simple denoising autoencoder training criterion is equivalent to matching the score (with respect to the data) of a specific energy-based model to that of a nonparametric Parzen density estimator of the data. This yields several useful insights. It defines a proper probabilistic model for the denoising autoencoder technique, which makes it in principle possible to sample from them or rank examples by their energy. It suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives. It justifies the use of tied weights between the encoder and decoder and suggests ways to extend the success of denoising autoencoders to a larger family of energy-based models. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> DAEs <s> Stacked denoising autoencoders (SDAs) have been successfully used to learn new representations for domain adaptation. Recently, they have attained record accuracy on standard benchmark tasks of sentiment analysis across different text domains. SDAs learn robust data representations by reconstruction, recovering original features from data that are artificially corrupted with noise. In this paper, we propose marginalized SDA (mSDA) that addresses two crucial limitations of SDAs: high computational cost and lack of scalability to high-dimensional features. In contrast to SDAs, our approach of mSDA marginalizes noise and thus does not require stochastic gradient descent or other optimization algorithms to learn parameters--in fact, they are computed in closed-form. Consequently, mSDA, which can be implemented in only 20 lines of MATLAB™, significantly speeds up SDAs by two orders of magnitude. Furthermore, the representations learnt by mSDA are as effective as the traditional SDAs, attaining almost identical accuracies in benchmark tasks. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> DAEs <s> What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments. <s> BIB006
Vincent et al. BIB002 , BIB003 proposed altering the training objective in BIB001 from mere reconstruction to that of denoising an artificially corrupted input, i.e., learning to reconstruct the clean input from a corrupted version. 14. Lðx; rÞ ¼ À P dx i¼1 x i logðr i Þ þ ð1 À r i Þ logð1 À r i Þ. 15. Contrary to traditional PCA loading factors, but similarly to the parameters learned by probabilistic PCA, the weight vectors learned by a linear autoencoder are not constrained to form an orthonormal basis nor to have a meaningful ordering. They will, however, span the same subspace. Learning the identity is no longer enough: The learner must capture the structure of the input distribution to optimally undo the effect of the corruption process, with the reconstruction essentially being a nearby but higher density point than the corrupted input. Fig. 3 illustrates that the DAE is learning a reconstruction function that corresponds to a vector field pointing toward high-density regions (the manifold where examples concentrate). Formally, the objective optimized by a DAE is where IE qðxjx ðtÞ Þ ½Á averages over corrupted examplesx drawn from corruption process qðx j x ðtÞ Þ. In practice, this is optimized by stochastic gradient descent, where the stochastic gradient is estimated by drawing one or a few corrupted versions of x ðtÞ each time x ðtÞ is considered. Corruptions considered in Vincent et al. BIB003 include additive isotropic Gaussian noise, salt and pepper noise for gray-scale images, and masking noise (salt or pepper only), for example, setting some randomly chosen inputs to 0 (independently per example). Masking noise has been used in most of the simulations. Qualitatively better features are reported with denoising, resulting in improved classification, and DAE features performed similarly or better than RBM features. Chen et al. BIB005 show that a simpler alternative with a closed-form solution can be obtained when restricting to a linear autoencoder and have successfully applied it to domain adaptation. Vincent BIB004 relates DAEs to energy-based probabilistic models: DAEs basically learn in rðxÞ Àx a vector pointing in the direction of the estimated score @ log pðxÞ @x ( Fig. 3 ). In the special case of linear reconstruction and squared error, Vincent BIB004 shows that training an affine-sigmoid-affine DAE amounts to learning an energy-based model whose energy function is very close to that of a GRBM. Training uses a regularized variant of the score matching parameter estimation technique  ,  ,  termed denoising score matching BIB004 . Swersky  had shown that training GRBMs with score matching is equivalent to training a regular autoencoder with an additional regularization term, while follow up on the theoretical results in BIB004 ,  showed the practical advantage of denoising to implement score matching efficiently. Finally, Alain and Bengio BIB006 generalize BIB004 and prove that DAEs of arbitrary parameterization with small Gaussian corruption noise are general estimators of the score.