Representation Learning: A Review and New Perspectives <s> Sampling Challenges <s> We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Sampling Challenges <s> It has been hypothesized, and supported with experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce Markov chains that mix faster between modes. Consequently, mixing between modes would be more efficient at higher levels of representation. To better understand this, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing between modes and interpolating between samples. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Sampling Challenges <s> The contractive auto-encoder learns a representation of the input data that captures the local manifold structure around each data point, through the leading singular vectors of the Jacobian of the transformation from input to representation. The corresponding singular values specify how much local variation is plausible in directions associated with the corresponding singular vectors, while remaining in a high-density region of the input space. This paper proposes a procedure for generating samples that are consistent with the local structure captured by a contractive auto-encoder. The associated stochastic process defines a distribution from which one can sample, and which experimentally appears to converge quickly and mix well between modes, compared to Restricted Boltzmann Machines and Deep Belief Networks. The intuitions behind this procedure can also be used to train the second layer of contraction that pools lower-level features and learns to be invariant to the local directions of variation discovered in the first layer. We show that this can help learn and represent invariances present in the data and improve classification error. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Sampling Challenges <s> Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of the unknown data generating density. This paper contributes to the mathematical understanding of this phenomenon and helps define better justified sampling algorithms for deep learning based on auto-encoder variants. We consider an MCMC where each step samples from a Gaussian whose mean and covariance matrix depend on the previous state, defines through its asymptotic distribution a target density. First, we show that good choices (in the sense of consistency) for these mean and covariance functions are the local expected value and local covariance under that target density. Then we show that an auto-encoder with a contractive penalty captures estimators of these local moments in its reconstruction function and its Jacobian. A contribution of this work is thus a novel alternative to maximum-likelihood density estimation, which we call local moment matching. It also justifies a recently proposed sampling algorithm for the Contractive Auto-Encoder and extends it to the Denoising Auto-Encoder. <s> BIB004
A troubling challenge with many probabilistic models with latent variables like most Boltzmann machine variants is that good MCMC sampling is required as part of the learning procedure, but that sampling becomes extremely inefficient (or unreliable) as training progresses because the modes of the learned distribution become sharper, making mixing between modes very slow. Whereas initially during training a learner assigns mass almost uniformly, as training progresse, its entropy decreases, approaching the entropy of the target distribution as more examples and more computation are provided. According to our manifold and natural clustering priors of Section 3.1, the target distribution has sharp modes (manifolds) separated by extremely low density areas. Mixing then becomes more difficult because MCMC methods, by their very nature, tend to make small steps to nearby high-probability configurations. This is illustrated in Fig. 7 . Bengio et al. BIB002 suggest that deep representations could help mixing between such well-separated modes, based on both theoretical arguments and on empirical evidence. The idea is that if higher level representations disentangle the underlying abstract factors better, then small steps in this abstract space (e.g., swapping from one category to another) can easily be done by MCMC. The high-level representations can then be mapped back to Fig. 6 . Sampling from regularized autoencoders BIB003 , BIB004 : Each MCMC step adds to current state x the noise , mostly in the direction of the estimated manifold tangent plane H and projects back toward the manifold (high-density regions) by performing a reconstruction step. Fig. 7 . Top: Early, during training, MCMC mixes easily between modes because the estimated distribution has high entropy and puts enough mass everywhere for small-steps movements (MCMC) to go from mode to mode. Bottom: Later on, training relying on good mixing can stall because estimated modes are separated by wide low-density deserts. the input space to obtain input-level samples, as in the DBNs sampling algorithm BIB001 . This has been demonstrated both with DBNs and with the newly proposed algorithm for sampling from contracting and DAEs BIB003 , BIB004 . This observation alone does not suffice to solve the problem of training a DBN or a DBM, but it may provide a crucial ingredient, and it makes it possible to consider successfully sampling from deep models trained by procedures that do not require an MCMC, like the stacked regularized autoencoders used in BIB003 .