Representation Learning: A Review and New Perspectives <s> Joint Training of DBMs <s> One of the central problems in machine learning and pattern recognition is to develop appropriate representations for complex data. We consider the problem of constructing a representation for data lying on a low-dimensional manifold embedded in a high-dimensional space. Drawing on the correspondence between the graph Laplacian, the Laplace Beltrami operator on the manifold, and the connections to the heat equation, we propose a geometrically motivated algorithm for representing the high-dimensional data. The algorithm provides a computationally efficient approach to nonlinear dimensionality reduction that has locality-preserving properties and a natural connection to clustering. Some potential applications and illustrative examples are discussed. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Joint Training of DBMs <s> e de MontrMontreal, Canada The family of decision tree learning algorithms is among the most widespread and studied. Motivated by the desire to develop learning algorithms that can generalize when learning highly varying functions such as those presumably needed to achieve artificial intelligence, we study some theoretical limitations of decision trees. We demonstrate formally that they can be seriously hurt by the curse of dimensionality in a sense that is a bit different from other nonparametric statistical methods, but most importantly, that they cannot generalize to variations not seen in the training set. This is because a decision tree creates a partition of the input space and needs at least one example in each of the regions associated with a leaf to make a sensible prediction in that region. A better understanding of the fundamental reasons for this limitation suggests that one should use forests or even deeper architectures instead of trees, which provide a form of distributed representation and can generalize to variations not encountered in the training data. <s> BIB002
We now consider the problem of joint training of all layers of a specific unsupervised model, the DBM. Whereas much progress (albeit with many unanswered questions) has been made on jointly training all the layers of deep architectures using back-propagated gradients (i.e., mostly in the supervised setting), much less work has been done on their purely unsupervised counterpart, for example, with DBMs. BIB002 Note, however, that one could hope that the successful techniques described in the previous section could be applied to unsupervised learning algorithms. Like the RBM, the DBM is another particular subset of the Boltzmann machine family of models where the units are again arranged in layers. However, unlike the RBM, the DBM possesses multiple layers of hidden units, with units in odd-numbered layers being conditionally independent given even-numbered layers, and vice versa. With respect to the Boltzmann energy function of BIB001 , the DBM corresponds to setting U ¼ 0 and a sparse connectivity structure in both V and W . We can make the structure of the DBM more explicit by specifying its energy function. For the model with two hidden layers, it is given as with ¼ fW; V ; d ð1Þ ; d ð2Þ ; bg. The DBM can also be characterized as a bipartite graph between two sets of vertices, formed by odd-and even-numbered layers (with v :¼ h ð0Þ ).