Representation Learning: A Review and New Perspectives <s> Leveraging the Modeled Tangent Spaces <s> In many machine learning applications, one has access, not only to training data, but also to some high-level a priori knowledge about the desired behavior of the system. For example, it is known in advance that the output of a character recognizer should be invariant with respect to small spatial distortions of the input images (translations, rotations, scale changes, etcetera). ::: ::: We have implemented a scheme that allows a network to learn the derivative of its outputs with respect to distortion operators of our choosing. This not only reduces the learning time and the amount of training data, but also provides a powerful language for specifying what generalizations we wish the network to perform. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Leveraging the Modeled Tangent Spaces <s> Memory-based classification algorithms such as radial basis functions or K-nearest neighbors typically rely on simple distances (Euclidean, dot product...), which are not particularly meaningful on pattern vectors. More complex, better suited distance measures are often expensive and rather ad-hoc (elastic matching, deformable templates). We propose a new distance measure which (a) can be made locally invariant to any set of transformations of the input and (b) can be computed efficiently. We tested the method on large handwritten character databases provided by the Post Office and the NIST. Using invariances with respect to translation, rotation, scaling, shearing and line thickness, the method consistently outperformed all other systems tested on the same databases. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Leveraging the Modeled Tangent Spaces <s> We present a series of theoretical arguments supporting the claim that a large class of modern learning algorithms that rely solely on the smoothness prior - with similarity between examples expressed with a local kernel - are sensitive to the curse of dimensionality, or more precisely to the variability of the target. Our discussion covers supervised, semi-supervised and unsupervised learning algorithms. These algorithms are found to be local in the sense that crucial properties of the learned function at x depend mostly on the neighbors of x in the training set. This makes them sensitive to the curse of dimensionality, well studied for classical non-parametric statistical learning. We show in the case of the Gaussian kernel that when the function to be learned has many variations, these algorithms require a number of training examples proportional to the number of variations, which could be large even though there may exist short descriptions of the target function, i.e. their Kolmogorov complexity may be low. This suggests that there exist non-local learning algorithms that at least have the potential to learn about such structured but apparently complex functions (because locally they have many variations), while not using very specific prior domain knowledge. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Leveraging the Modeled Tangent Spaces <s> We combine three important ideas present in previous work for building classifiers: the semi-supervised hypothesis (the input distribution contains information about the classifier), the unsupervised manifold hypothesis (data density concentrates near low-dimensional manifolds), and the manifold hypothesis for classification (different classes correspond to disjoint manifolds separated by low density). We exploit a novel algorithm for capturing manifold structure (high-order contractive auto-encoders) and we show how it builds a topological atlas of charts, each chart being characterized by the principal singular vectors of the Jacobian of a representation mapping. This representation learning algorithm can be stacked to yield a deep architecture, and we combine it with a domain knowledge-free version of the TangentProp algorithm to encourage the classifier to be insensitive to local directions changes along the manifold. Record-breaking classification results are obtained. <s> BIB004
The local tangent space, at a point along the manifold, can be thought of as capturing locally valid transformations that were prominent in the training data. For example, Rifai et al. BIB004 examine the tangent directions extracted with an SVD of the Jacobian of CAEs trained on digits, images, or text-document data: They appear to correspond to small translations or rotations for images or digits and to substitutions of words within the same theme for documents. Such very local transformations along a data manifold are not expected to change class identity. To build their MTC, Rifai et al. BIB004 then apply techniques such as tangent distance BIB002 and tangent propagation BIB001 , which were initially developed to build classifiers that are insensitive to input deformations provided as prior domain knowledge. Now, these techniques are applied using the local leading tangent directions extracted by a CAE, i.e., not using any prior domain knowledge (except the broad prior about the existence of a manifold). This approach set a new record for MNIST digit classification among prior-knowledge free approaches. BIB003 