Representation Learning: A Review and New Perspectives <s> 5. <s> We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent expectations are estimated using a variational approximation that tends to focus on a single mode, and dataindependent expectations are approximated using persistent Markov chains. The use of two quite different techniques for estimating the two types of expectation that enter into the gradient of the log-likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass. We present results on the MNIST and NORB datasets showing that deep Boltzmann machines learn good generative models and perform well on handwritten digit and visual object recognition tasks. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> 5. <s> The deep Boltzmann machine (DBM) has been an important development in the quest for powerful "deep" probabilistic models. To date, simultaneous or joint training of all layers of the DBM has been largely unsuccessful with existing training methods. We introduce a simple regularization scheme that encourages the weight vectors associated with each hidden unit to have similar norms. We demonstrate that this regularization can be easily combined with standard stochastic maximum likelihood to yield an effective training strategy for the simultaneous training of all layers of the deep Boltzmann machine. <s> BIB002
Finally, update the model parameters with a step of approximate stochastic gradient ascent. While the above procedure appears to be a simple extension of the highly effective SML scheme for training RBMs, as we demonstrate in Desjardins et al. BIB002 , this procedure seems vulnerable to falling in poor local minima that leave many hidden units effectively dead (not significantly different from its random initialization with small norm). The failure of the SML joint training strategy was noted by Salakhutdinov and Hinton BIB001 . As an alternative, they proposed a greedy layerwise training strategy. This procedure consists of pretraining the layers of the DBM, in much the same way as the DBN, i.e., by stacking RBMs and training each layer to independently model the output of the previous layer. A final joint "fine tuning" is done following the above SML-based procedure.