Representation Learning: A Review and New Perspectives <s> Undirected Graphical Models <s> Abstract : At this early stage in the development of cognitive science, methodological issues are both open and central. There may have been times when developments in neuroscience, artificial intelligence, or cognitive psychology seduced researchers into believing that their discipline was on the verge of discovering the secret of intelligence. But a humbling history of hopes disappointed has produced the realization that understanding the mind will challenge the power of all these methodologies combined. The work reported in this chapter rests on the conviction that a methodology that has a crucial role to play in the development of cognitive science is mathematical analysis. The success of cognitive science, like that of many other sciences, will, I believe, depend upon the construction of a solid body of theoretical results: results that express in a mathematical language the conceptual insights of the field; results that squeeze all possible implications out of those insights by exploiting powerful mathematical techniques. This body of results, which I will call the theory of information processing, exists because information is a concept that lends itself to mathematical formalization. One part of the theory of information processing is already well-developed. The classical theory of computation provides powerful and elegant results about the notion of effective procedure, including languages for precisely expressing them and theoretical machines for realizing them. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Undirected Graphical Models <s> One of the central problems in machine learning and pattern recognition is to develop appropriate representations for complex data. We consider the problem of constructing a representation for data lying on a low-dimensional manifold embedded in a high-dimensional space. Drawing on the correspondence between the graph Laplacian, the Laplace Beltrami operator on the manifold, and the connections to the heat equation, we propose a geometrically motivated algorithm for representing the high-dimensional data. The algorithm provides a computationally efficient approach to nonlinear dimensionality reduction that has locality-preserving properties and a natural connection to clustering. Some potential applications and illustrative examples are discussed. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Undirected Graphical Models <s> Most of the existing approaches to collaborative filtering cannot handle very large data sets. In this paper we show how a class of two-layer undirected graphical models, called Restricted Boltzmann Machines (RBM's), can be used to model tabular data, such as user's ratings of movies. We present efficient learning and inference procedures for this class of models and demonstrate that RBM's can be successfully applied to the Netflix data set, containing over 100 million user/movie ratings. We also show that RBM's slightly outperform carefully-tuned SVD models. When the predictions of multiple RBM models and multiple SVD models are linearly combined, we achieve an error rate that is well over 6% better than the score of Netflix's own system. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Undirected Graphical Models <s> The Conditional Restricted Boltzmann Machine (CRBM) is a recently proposed model for time series that has a rich, distributed hidden state and permits simple, exact inference. We present a new model, based on the CRBM that preserves its most important computational properties and includes multiplicative three-way interactions that allow the effective interaction weight between two units to be modulated by the dynamic state of a third unit. We factor the three-way weight tensor implied by the multiplicative model, reducing the number of parameters from O(N3) to O(N2). The result is an efficient, compact model whose effectiveness we demonstrate by modeling human motion. Like the CRBM, our model can capture diverse styles of motion with a single set of parameters, and the three-way interactions greatly improve the model's ability to blend motion styles or to transition smoothly among them. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> Undirected Graphical Models <s> To allow the hidden units of a restricted Boltzmann machine to model the transformation between two successive images, Memisevic and Hinton (2007) introduced three-way multiplicative interactions that use the intensity of a pixel in the first image as a multiplicative gain on a learned, symmetric weight between a pixel in the second image and a hidden unit. This creates cubically many parameters, which form a three-dimensional interaction tensor. We describe a low-rank approximation to this interaction tensor that uses a sum of factors, each of which is a three-way outer product. This approximation allows efficient learning of transformations between larger image patches. Since each factor can be viewed as an image filter, the model as a whole learns optimal filter pairs for efficiently representing transformations. We demonstrate the learning of optimal filter pairs from various synthetic and real image sequences. We also show how learning about image transformations allows the model to perform a simple visual analogy task, and we show how a completely unsupervised network trained on transformations perceives multiple motions of transparent dot patterns in the same way as humans. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> Undirected Graphical Models <s> Learning a generative model of natural images is a useful way of extracting features that capture interesting regularities. Previous work on learning such models has focused on methods in which the latent features are used to determine the mean and variance of each pixel independently, or on methods in which the hidden units determine the covariance matrix of a zero-mean Gaussian distribution. In this work, we propose a probabilistic model that combines these two approaches into a single framework. We represent each image using one set of binary latent features that model the image-specific covariance and a separate set that model the mean. We show that this approach provides a probabilistic framework for the widely used simple-cell complex-cell architecture, it produces very realistic samples of natural images and it extracts features that yield state-of-the-art recognition accuracy on the challenging CIFAR 10 dataset. <s> BIB006
Undirected graphical models, also called Markov random fields (MRFs), parameterize the joint pðx; hÞ through a product of unnormalized nonnegative clique potentials: where i ðxÞ, j ðhÞ, and k ðx; hÞ are the clique potentials describing the interactions between the visible elements, between the hidden variables, and those interaction between the visible and hidden variables, respectively. The partition function Z ensures that the distribution is normalized. Within the context of unsupervised feature learning, we generally see a particular form of MRF called a Boltzmann distribution with clique potentials constrained to be positive: where E ðx; hÞ is the energy function and contains the interactions described by the MRF clique potentials and are the model parameters that characterize these interactions. The Boltzmann machine was originally defined as a network of symmetrically coupled binary random variables or units. These stochastic units can be divided into two groups: 1) the visible units x 2 f0; 1g dx that represent the data, and 2) the hidden or latent units h 2 f0; 1g d h that mediate dependencies between the visible units through their mutual interactions. The pattern of interaction is specified through the energy function: where ¼ fU; V ; W ; b; dg are the model parameters that, respectively, encode the visible-to-visible interactions, the hidden-to-hidden interactions, the visible-to-hidden interactions, the visible self-connections, and the hidden selfconnections (called biases). To avoid overparameterization, the diagonals of U and V are set to zero. The Boltzmann machine energy function specifies the probability distribution over ½x; h, via the Boltzmann distribution,  , with the partition function Z given by This joint probability distribution gives rise to the set of conditional distributions of the form: In general, inference in the Boltzmann machine is intractable. For example, computing the conditional probability of h i given the visibles, P ðh i j xÞ, requires marginalizing over the rest of the hiddens, which implies evaluating a sum with 2 dhÀ1 terms: However, with some judicious choices in the pattern of interactions between the visible and hidden units, more tractable subsets of the model family are possible, as we discuss next. RBMs. The RBM is likely the most popular subclass of Boltzmann machine BIB001 . It is defined by restricting the interactions in the Boltzmann energy function, in BIB002 , to only those between h and x, i.e., E RBM is E BM with U ¼ 0 and V ¼ 0. As such, the RBM can be said to form a bipartite graph with the visibles and the hiddens forming two layers of vertices in the graph (and no connection between units of the same layer). With this restriction, the RBM possesses the useful property that the conditional distribution over the hidden units factorizes given the visibles: Likewise, the conditional distribution over the visible units given the hiddens also factorizes: This makes inferences readily tractable in RBMs. For example, the RBM feature representation is taken to be the set of posterior marginals P ðh i j xÞ which, given the conditional independence described in  , are immediately available. Note that this is in stark contrast to the situation with popular directed graphical models for unsupervised feature extraction, where computing the posterior probability is intractable. Importantly, the tractability of the RBM does not extend to its partition function, which still involves summing an exponential number of terms. It does imply, however, that we can limit the number of terms to minf2 d x ; 2 d h g. Usually, this is still an unmanageable number of terms, and therefore we must resort to approximate methods to deal with its estimation. It is difficult to overstate the impact the RBM has had to the fields of unsupervised feature learning and deep learning. It has been used in a truly impressive variety of applications, including fMRI image classification  , motion and spatial transformations BIB004 , BIB005 , collaborative filtering BIB003 , and natural image modeling BIB006 ,  .