Representation Learning: A Review and New Perspectives <s> Mean-Field Approximate Inference <s> One approach to invariant object recognition employs a recurrent neural network as an associative memory. In the standard depiction of the network's state space, memories of objects are stored as attractive fixed points of the dynamics. I argue for a modification of this picture: if an object has a continuous family of instantiations, it should be represented by a continuous attractor. This idea is illustrated with a network that learns to complete patterns. To perform the task of filling in missing information, the network develops a continuous attractor that models the manifold from which the patterns are drawn. From a statistical view-point, the pattern completion task allows a formulation of unsupervised learning in terms of regression rather than density estimation. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Mean-Field Approximate Inference <s> We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent expectations are estimated using a variational approximation that tends to focus on a single mode, and dataindependent expectations are approximated using persistent Markov chains. The use of two quite different techniques for estimating the two types of expectation that enter into the gradient of the log-likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass. We present results on the MNIST and NORB datasets showing that deep Boltzmann machines learn good generative models and perform well on handwritten digit and visual object recognition tasks. <s> BIB002
A key point of departure from the RBM is that the posterior distribution over the hidden units (given the visibles) is no longer tractable due to the interactions between the hidden units. Salakhutdinov and Hinton BIB002 resort to a meanfield approximation to the posterior. Specifically, in the case of a model with two hidden layers, we wish to approximate P ðh ð1Þ ; h ð2Þ j vÞ with the factored distribution Q v ðh ð1Þ ; h ð2Þ Þ ¼ i Þ such that the KL divergence KLðP ðh ð1Þ ; h ð2Þ j vÞkQ v ðh 1 ; h 2 ÞÞ is minimized or, equivalently, that a lower bound to the log likelihood is maximized:  Maximizing this lower bound with respect to the meanfield distribution Q v ðh 1 ; h 2 Þ (by setting derivatives to zero) yields the following mean field update equations: Note how the above equations ostensibly look like a fixed point recurrent neural network, i.e., with constant input. In the same way that an RBM can be associated with a simple autoencoder, the above mean-field update equations for the DBM can be associated with a recurrent autoencoder. In that case, the training criterion involves the reconstruction error at the last or at consecutive time steps. This type of model has been explored by Savard  and Seung BIB001 and shown to do a better job at denoising than ordinary autoencoders. Iterating (31-32) until convergence yields the Q parameters of the "variational positive phase" of  This variational learning procedure leaves the "negative phase" untouched, which can thus be estimated through SML or CD  as in the RBM case.