Representation Learning: A Review and New Perspectives <s> REPRESENTATION LEARNING AS MANIFOLD LEARNING <s> Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered. > <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> REPRESENTATION LEARNING AS MANIFOLD LEARNING <s> A new method for performing a nonlinear form of principal component analysis is proposed. By the use of integral operator kernel functions, one can efficiently compute principal components in high-dimensional feature spaces, related to input space by some nonlinear map—for instance, the space of all possible five-pixel products in 16 × 16 images. We give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> REPRESENTATION LEARNING AS MANIFOLD LEARNING <s> The similarity between objects is a fundamental element of many learning algorithms. Most non-parametric methods take this similarity to be fixed, but much recent work has shown the advantages of learning it, in particular to exploit the local invariances in the data or to capture the possibly non-linear manifold on which most of the data lies. We propose a new non-parametric kernel density estimation method which captures the local structure of an underlying manifold through the leading eigenvectors of regularized local covariance matrices. Experiments in density estimation show significant improvements with respect to Parzen density estimators. The density estimators can also be used within Bayes classifiers, yielding classification rates similar to SVMs and much superior to the Parzen classifier. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> REPRESENTATION LEARNING AS MANIFOLD LEARNING <s> We construct a nonlinear mapping from a high-dimensional sample space to a low-dimensional vector space, effectively recovering a Cartesian coordinate system for the manifold from which the data is sampled. The mapping preserves local geometric relations in the manifold and is pseudo-invertible. We show how to estimate the intrinsic dimensionality of the manifold from samples, decompose the sample data into locally linear low-dimensional patches, merge these patches into a single low-dimensional coordinate system, and compute forward and reverse mappings between the sample and coordinate spaces. The objective functions are convex and their solutions are given in closed form. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> REPRESENTATION LEARNING AS MANIFOLD LEARNING <s> We describe a probabilistic approach to the task of placing objects, described by high-dimensional vectors or by pairwise dissimilarities, in a low-dimensional space in a way that preserves neighbor identities. A Gaussian is centered on each object in the high-dimensional space and the densities under this Gaussian (or the given dissimilarities) are used to define a probability distribution over all the potential neighbors of the object. The aim of the embedding is to approximate this distribution as well as possible when the same operation is performed on the low-dimensional "images" of the objects. A natural cost function is a sum of Kullback-Leibler divergences, one per object, which leads to a simple gradient for adjusting the positions of the low-dimensional images. Unlike other dimensionality reduction methods, this probabilistic framework makes it easy to represent each object by a mixture of widely separated low-dimensional images. This allows ambiguous objects, like the document count vector for the word "bank", to have versions close to the images of both "river" and "finance" without forcing the images of outdoor concepts to be located close to those of corporate concepts. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> REPRESENTATION LEARNING AS MANIFOLD LEARNING <s> One of the central problems in machine learning and pattern recognition is to develop appropriate representations for complex data. We consider the problem of constructing a representation for data lying on a low-dimensional manifold embedded in a high-dimensional space. Drawing on the correspondence between the graph Laplacian, the Laplace Beltrami operator on the manifold, and the connections to the heat equation, we propose a geometrically motivated algorithm for representing the high-dimensional data. The algorithm provides a computationally efficient approach to nonlinear dimensionality reduction that has locality-preserving properties and a natural connection to clustering. Some potential applications and illustrative examples are discussed. <s> BIB006 </s> Representation Learning: A Review and New Perspectives <s> REPRESENTATION LEARNING AS MANIFOLD LEARNING <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB007 </s> Representation Learning: A Review and New Perspectives <s> REPRESENTATION LEARNING AS MANIFOLD LEARNING <s> Several unsupervised learning algorithms based on an eigendecomposition provide either an embedding or a clustering only for given training points, with no straightforward extension for out-of-sample examples short of recomputing eigenvectors. This paper provides a unified framework for extending Local Linear Embedding (LLE), Isomap, Laplacian Eigenmaps, Multi-Dimensional Scaling (for dimensionality reduction) as well as for Spectral Clustering. This framework is based on seeing these algorithms as learning eigenfunctions of a data-dependent kernel. Numerical experiments show that the generalizations performed have a level of error comparable to the variability of the embedding algorithms due to the choice of training data. <s> BIB008 </s> Representation Learning: A Review and New Perspectives <s> REPRESENTATION LEARNING AS MANIFOLD LEARNING <s> Scientists working with large volumes of high-dimensional data, such as global climate patterns, stellar spectra, or human gene distributions, regularly confront the problem of dimensionality reduction: finding meaningful low-dimensional structures hidden in their high-dimensional observations. The human brain confronts the same problem in everyday perception, extracting from its high-dimensional sensory inputs—30,000 auditory nerve fibers or 106 optic nerve fibers—a manageably small number of perceptually relevant features. Here we describe an approach to solving dimensionality reduction problems that uses easily measured local metric information to learn the underlying global geometry of a data set. Unlike classical techniques such as principal component analysis (PCA) and multidimensional scaling (MDS), our approach is capable of discovering the nonlinear degrees of freedom that underlie complex natural observations, such as human handwriting or images of a face under different viewing conditions. In contrast to previous algorithms for nonlinear dimensionality reduction, ours efficiently computes a globally optimal solution, and, for an important class of data manifolds, is guaranteed to converge asymptotically to the true structure. <s> BIB009
Another important perspective on representation learning is based on the geometric notion of manifold. Its premise is the manifold hypothesis, according to which real-world data presented in high-dimensional spaces are expected to concentrate in the vicinity of a manifold M of much lower dimensionality d M , embedded in high-dimensional input space IR dx . This prior seems particularly well suited for AI tasks such as those involving images, sounds, or text, for which most uniformly sampled input configurations are unlike natural stimuli. As soon as there is a notion of "representation," one can think of a manifold by considering the variations in input space which are captured by or reflected (by corresponding changes) in the learned representation. To first approximation, some directions are well preserved (the tangent directions of the manifold), while others are not (directions orthogonal to the manifolds). With this perspective, the primary unsupervised learning task is then seen as modeling the structure of the data-supporting manifold. BIB001 The associated representation being learned can be associated with an intrinsic coordinate system on the embedded manifold. The archetypal manifold modeling algorithm is, not surprisingly, also the archetypal lowdimensional representation learning algorithm: PCA, which models a linear manifold. It was initially devised with the objective of finding the closest linear manifold to a cloud of data points. The principal components, i.e., the representation f ðxÞ that PCA yields for an input point x, uniquely locates its projection on that manifold: It corresponds to intrinsic coordinates on the manifold. Data manifolds for complex real-world domains are, however, expected to be strongly nonlinear. Their modeling is sometimes approached as patchworks of locally linear tangent spaces BIB003 , BIB004 . The large majority of algorithms built on this geometric perspective adopt a nonparametric approach, based on a training set nearest neighbor graph BIB002 ,  , BIB009 , BIB004 , BIB006 ,  ,  , BIB005 ,  . In these nonparametric approaches, each high-dimensional training point has its own set of free low-dimensional embedding coordinates, which are optimized so that certain properties of the neighborhood graph computed in original high-dimensional input space are best preserved. These methods, however, do not directly learn a parameterized feature extraction function f ðxÞ applicable to new test points, BIB007 which seriously limits their use as feature extractors, except in a transductive setting. Comparatively, few nonlinear manifold learning methods have been proposed that learn a parametric map that can directly compute a representation for new points; we will focus on these. BIB001 . Actually, data points need not strictly lie on the "manifold," but the probability density is expected to fall off sharply as one moves away from it, and it may actually be constituted of several possibly disconnected manifolds with different intrinsic dimensionality. BIB007 . For several of these techniques, representations for new points can be computed using the Nyströ m approximation as has been proposed as an extension in BIB008 , but this remains cumbersome and computationally expensive.