Representation Learning: A Review and New Perspectives <s> Learning a Parametric Mapping Based on a Neighborhood Graph <s> Several unsupervised learning algorithms based on an eigendecomposition provide either an embedding or a clustering only for given training points, with no straightforward extension for out-of-sample examples short of recomputing eigenvectors. This paper provides a unified framework for extending Local Linear Embedding (LLE), Isomap, Laplacian Eigenmaps, Multi-Dimensional Scaling (for dimensionality reduction) as well as for Spectral Clustering. This framework is based on seeing these algorithms as learning eigenfunctions of a data-dependent kernel. Numerical experiments show that the generalizations performed have a level of error comparable to the variability of the embedding algorithms due to the choice of training data. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Learning a Parametric Mapping Based on a Neighborhood Graph <s> We claim and present arguments to the effect that a large class of manifold learning algorithms that are essentially local and can be framed as kernel learning algorithms will suffer from the curse of dimensionality, at the dimension of the true underlying manifold. This observation suggests to explore non-local manifold learning algorithms which attempt to discover shared structure in the tangent planes at different positions. A criterion for such an algorithm is proposed and experiments estimating a tangent plane prediction function are presented, showing its advantages with respect to local manifold learning algorithms: it is able to generalize very far from training data (on learning handwritten character image rotations), where a local non-parametric method fails. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Learning a Parametric Mapping Based on a Neighborhood Graph <s> To escape from the curse of dimensionality, we claim that one can learn non-local functions, in the sense that the value and shape of the learned function at x must be inferred using examples that may be far from x. With this objective, we present a non-local non-parametric density estimator. It builds upon previously proposed Gaussian mixture models with regularized covariance matrices to take into account the local shape of the manifold. It also builds upon recent work on non-local estimators of the tangent plane of a manifold, which are able to generalize in places with little training data, unlike traditional, local, non-parametric models. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Learning a Parametric Mapping Based on a Neighborhood Graph <s> We show how nonlinear embedding algorithms popular for use with shallow semi-supervised learning techniques such as kernel methods can be applied to deep multilayer architectures, either as a regularizer at the output layer, or on each layer of the architecture. This provides a simple alternative to existing approaches to deep learning whilst yielding competitive error rates compared to those methods, and existing shallow semi-supervised techniques. <s> BIB004
Some of the above nonparametric manifold learning algorithms can be modified to learn a parametric mapping f , i.e., applicable to new points: Instead of having free lowdimensional embedding coordinate "parameters" for each training point, these coordinates are obtained through an explicitly parameterized function, as with the parametric variant  of t-SNE  . Instead, semi-supervised embedding BIB004 learns a direct encoding while taking into account the manifold hypothesis through a neighborhood graph. A parameterized neural network architecture simultaneously learns a manifold embedding and a classifier. The training criterion encourages training set neigbhors to have similar representations. The reduced and tightly controlled number of free parameters in such parametric methods, compared to their pure nonparametric counterparts, forces models to generalize the manifold shape nonlocally BIB003 , which can translate into better features and final performance  . However, basing the modeling of manifolds on training set neighborhood relationships might be risky statistically in highdimensional spaces (sparsely populated due to the curse of dimensionality) as, for example, most euclidean nearest neighbors risk having too little in common semantically. The nearest neighbor graph is simply not sufficiently densely populated to map out satisfyingly the wrinkles of the target manifold BIB002 , BIB003 ,  . It can also become problematic computationally to consider all pairs of data points, BIB001 which scales quadratically with training set size.