Representation Learning: A Review and New Perspectives <s> CAEs <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> CAEs <s> We claim and present arguments to the effect that a large class of manifold learning algorithms that are essentially local and can be framed as kernel learning algorithms will suffer from the curse of dimensionality, at the dimension of the true underlying manifold. This observation suggests to explore non-local manifold learning algorithms which attempt to discover shared structure in the tangent planes at different positions. A criterion for such an algorithm is proposed and experiments estimating a tangent plane prediction function are presented, showing its advantages with respect to local manifold learning algorithms: it is able to generalize very far from training data (on learning handwritten character image rotations), where a local non-parametric method fails. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> CAEs <s> We show how to learn a deep graphical model of the word-count vectors obtained from a large set of documents. The values of the latent variables in the deepest layer are easy to infer and give a much better representation of each document than Latent Semantic Analysis. When the deepest layer is forced to use a small number of binary variables (e.g. 32), the graphical model performs ''semantic hashing'': Documents are mapped to memory addresses in such a way that semantically similar documents are located at nearby addresses. Documents similar to a query document can then be found by simply accessing all the addresses that differ by only a few bits from the address of the query document. This way of extending the efficiency of hash-coding to approximate matching is much faster than locality sensitive hashing, which is the fastest current method. By using semantic hashing to filter the documents given to TF-IDF, we achieve higher accuracy than applying TF-IDF to the entire document set. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> CAEs <s> We present in this paper a novel approach for training deterministic auto-encoders. We show that by adding a well chosen penalty term to the classical reconstruction cost function, we can achieve results that equal or surpass those attained by other regularized auto-encoders as well as denoising auto-encoders on a range of datasets. This penalty term corresponds to the Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input. We show that this penalty term results in a localized space contraction which in turn yields robust features on the activation layer. Furthermore, we show how this penalty term is related to both regularized auto-encoders and denoising auto-encoders and how it can be seen as a link between deterministic and non-deterministic auto-encoders. We find empirically that this penalty helps to carve a representation that better captures the local directions of variation dictated by the data, corresponding to a lower-dimensional non-linear manifold, while being more invariant to the vast majority of directions orthogonal to the manifold. Finally, we show that by using the learned features to initialize a MLP, we achieve state of the art classification error on a range of datasets, surpassing other methods of pretraining. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> CAEs <s> We propose a novel regularizer when training an autoencoder for unsupervised feature extraction. We explicitly encourage the latent representation to contract the input space by regularizing the norm of the Jacobian (analytically) and the Hessian (stochastically) of the encoder's output with respect to its input, at the training points. While the penalty on the Jacobian's norm ensures robustness to tiny corruption of samples in the input space, constraining the norm of the Hessian extends this robustness when moving further away from the sample. From a manifold learning perspective, balancing this regularization with the auto-encoder's reconstruction objective yields a representation that varies most when moving along the data manifold in input space, and is most insensitive in directions orthogonal to the manifold. The second order regularization, using the Hessian, penalizes curvature, and thus favors smooth manifold. We show that our proposed technique, while remaining computationally efficient, yields representations that are significantly better suited for initializing deep architectures than previously proposed approaches, beating state-of-the-art performance on a number of datasets. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> CAEs <s> What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments. <s> BIB006
CAEs, proposed by Rifai et al. BIB004 , follow up on DAEs and share a similar motivation of learning robust representations. CAEs achieve this by adding an analytic contractive penalty to BIB001 , the Frobenius norm of the encoder's Jacobian, and results in penalizing the sensitivity of learned features to infinitesimal input variations. Let JðxÞ ¼ @f @x ðxÞ be the Jacobian matrix of the encoder at x. The CAE's training objective is where is a hyperparameter controlling the strength of the regularization. For an affine sigmoid encoder, the contractive penalty term is easy to compute: There are at least three notable differences with DAEs, which may be partly responsible for the better performance that CAE features seem to empirically demonstrate: 1) The sensitivity of the features is penalized  rather than that of the reconstruction. 2) The penalty is analytic rather than stochastic: An efficiently computable expression replaces what might otherwise require d x corrupted samples to size up (i.e., the sensitivity in d x directions). 3) A hyperparameter allows fine control of the tradeoff between reconstruction and robustness (while the two are mingled in a DAE). Note, however, that there is a tight connection between the DAE and the CAE: As shown in BIB006 , a DAE with small corruption noise can be seen (through a Taylor expansion) as a type of contractive autoencoder where the contractive penalty is on the whole reconstruction function rather than just on the encoder. BIB002 A potential disadvantage of the CAE's analytic penalty is that it amounts to only encouraging robustness to infinitesimal input variations. This is remedied in BIB005 with the CAE+H, which penalizes all higher order derivatives in an efficient stochastic manner by adding a term that encourages JðxÞ and Jðx þ Þ to be close: where $ N ð0; 2 IÞ, and is the associated regularization strength hyperparameter. The DAE and CAE have been successfully used to win the final phase of the Unsupervised and Transfer Learning Challenge  . The representation learned by the CAE Fig. 3 . When data concentrate near a lower dimensional manifold, the corruption vector is typically almost orthogonal to the manifold, and the reconstruction function learns to denoise, map from low-probability configurations (corrupted inputs) to high-probability ones (original inputs), creating a vector field aligned with the score (derivative of the estimated density).  . That is, the robustness of the representation is encouraged. BIB002 . But note that in the CAE, the decoder weights are tied to the encoder weights to avoid degenerate solutions, and this should also make the decoder contractive. tends to be saturated rather than sparse, i.e., most hidden units are near the extremes of their range (e.g., 0 or 1), and their derivative @hiðxÞ @x is near 0. The nonsaturated units are few and sensitive to the inputs, with their associated filters (weight vectors) together forming a basis explaining the local changes around x, as discussed in Section 8.2. Another way to get saturated (nearly binary) units is semantic hashing BIB003 .