Representation Learning: A Review and New Perspectives <s> RBM Parameter Estimation <s> We present a distribution model for binary vectors, called the influence combination model and show how this model can be used as the basis for unsupervised learning algorithms for feature selection. The model can be represented by a particular type of Boltzmann machine with a bipartite graph structure that we call the combination machine. This machine is closely related to the Harmonium model defined by Smolensky. In the first part of the paper we analyze properties of this distribution representation scheme. We show that arbitrary distributions of binary vectors can be approximated by the combination model. We show how the weight vectors in the model can be interpreted as high order correlation patterns among the input bits, and how the combination machine can be used as a mechanism for detecting these patterns. We compare the combination model with the mixture model and with principle component analysis. In the second part of the paper we present two algorithms for learning the combination model from examples. The first learning algorithm is the standard gradient ascent heuristic for computing maximum likelihood estimates for the parameters of the model. Here we give a closed form for this gradient that is significantly easier to compute than the corresponding gradient for the general Boltzmann machine. The second learning algorithm is a greedy method that creates the hidden units and computes their weights one at a time. This method is a variant of projection pursuit density estimation. In the third part of the paper we give experimental results for these learning methods on synthetic data and on natural data of handwritten digit images. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> RBM Parameter Estimation <s> We analyse the convergence of stochastic algorithms with Markovian noise when the ergodicity of the Markov chain governing the noise rapidly decreases as the control parameter tends to infinity. In such a case, there may be a positive probability of divergence of the algorithm in the classic Robbins-Monro form. We provide sufficient condition which ensure convergence. Moreover, we analyse the asymptotic behaviour of these algorithms and state a diffusion approximation theorem <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> RBM Parameter Estimation <s> We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> RBM Parameter Estimation <s> A new algorithm for training Restricted Boltzmann Machines is introduced. The algorithm, named Persistent Contrastive Divergence, is different from the standard Contrastive Divergence algorithms in that it aims to draw samples from almost exactly the model distribution. It is compared to some standard Contrastive Divergence and Pseudo-Likelihood algorithms on the tasks of modeling and classifying various types of data. The Persistent Contrastive Divergence algorithm outperforms the other algorithms, and is equally fast and simple. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> RBM Parameter Estimation <s> The most commonly used learning algorithm for restricted Boltzmann machines is contrastive divergence which starts a Markov chain at a data point and runs the chain for only a few iterations to get a cheap, low variance estimate of the sufficient statistics under the model. Tieleman (2008) showed that better learning can be achieved by estimating the model's statistics using a small set of persistent "fantasy particles" that are not reinitialized to data points after each weight update. With sufficiently small weight updates, the fantasy particles represent the equilibrium distribution accurately but to explain why the method works with much larger weight updates it is necessary to consider the interaction between the weight updates and the Markov chain. We show that the weight updates force the Markov chain to mix fast, and using this insight we develop an even faster mixing chain that uses an auxiliary set of "fast weights" to implement a temporary overlay on the energy landscape. The fast weights learn rapidly but also decay rapidly and do not contribute to the normal energy landscape that defines the model. <s> BIB005
Many of the RBM training methods we discuss here are applicable to more general undirected graphical models, but are particularly practical in the RBM setting. Freund and Haussler BIB001 proposed a learning algorithm for harmoniums (RBMs) based on projection pursuit. CD  , BIB003 has been used most often to train RBMs, and many recent papers use stochastic maximum likelihood (SML) BIB002 , BIB004 . As discussed in Section 6.1, in training probabilistic models, parameters are typically adapted to maximize the likelihood of the training data (or equivalently the log likelihood, or its penalized version, which adds a regularization term). With T training examples, the log likelihood is given by Gradient-based optimization requires its gradient, which for Boltzmann machines is given by where we have the expectations with respect to pðh ðtÞ j x ðtÞ Þ under the "clamped" condition (also called the positive phase) and over the full joint pðx; hÞ under the "unclamped" condition (also called the negative phase). Intuitively, the gradient acts to locally move the model distribution (the negative phase distribution) toward the data distribution (positive phase distribution) by pushing down the energy of ðh; x ðtÞ Þ pairs (for h $ P ðh j x ðtÞ Þ) while pushing up the energy of ðh; xÞ pairs (for ðh; xÞ $ P ðh; xÞ) until the two forces are in equilibrium, at which point the sufficient statistics (gradient of the energy function) have equal expectations with x sampled from the training distribution or with x sampled from the model. The RBM conditional independence properties imply that the expectation in the positive phase of (15) is tractable. The negative phase term-arising from the partition function's contribution to the log-likelihood gradient-is more problematic because the computation of the expectation over the joint is not tractable. The various ways of dealing with the partition function's contribution to the gradient have brought about a number of different training algorithms, many trying to approximate the log-likelihood gradient. To approximate the expectation of the joint distribution in the negative phase contribution to the gradient, it is natural to again consider exploiting the conditional independence of the RBM to specify a Monte Carlo approximation of the expectation over the joint: with the samples ðx ðlÞ ;h ðlÞ Þ drawn by a block Gibbs Markov chain Monte Carlo (MCMC) sampling procedure: x ðlÞ $ P ðx jh ðlÀ1Þ Þ; h ðlÞ $ P ðh jx ðlÞ Þ: Naively, for each gradient update step one would start a Gibbs sampling chain, wait until the chain converges to the equilibrium distribution, and then draw a sufficient number of samples to approximate the expected gradient with respect to the model (joint) distribution in  . Then, restart the process for the next step of approximate gradient ascent on the log-likelihood. This procedure has the obvious flaw that waiting for the Gibbs chain to "burn-in" and reach equilibrium anew for each gradient update cannot form the basis of a practical training algorithm. CD  , BIB003 , SML BIB002 , BIB004 , and fast-weights persistent CD or fast-weights persistent contrastive divergence (FPCD) BIB005 are all ways to avoid or reduce the need for burn in.