Representation Learning: A Review and New Perspectives <s> Algorithms to Disentangle Factors of Variation <s> The artificial neural networks that are used to recognize shapes typically use one or more layers of learned feature detectors that produce scalar outputs. By contrast, the computer vision community uses complicated, hand-engineered features, like SIFT [6], that produce a whole vector of outputs including an explicit representation of the pose of the feature. We show how neural networks can be used to learn features that output a whole vector of instantiation parameters and we argue that this is a much more promising way of dealing with variations in position, orientation, scale and lighting than the methods currently employed in the neural networks community. It is also more promising than the hand-engineered features currently used in computer vision because it provides an efficient way of adapting the features to the domain. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Algorithms to Disentangle Factors of Variation <s> Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of the unknown data generating density. This paper contributes to the mathematical understanding of this phenomenon and helps define better justified sampling algorithms for deep learning based on auto-encoder variants. We consider an MCMC where each step samples from a Gaussian whose mean and covariance matrix depend on the previous state, defines through its asymptotic distribution a target density. First, we show that good choices (in the sense of consistency) for these mean and covariance functions are the local expected value and local covariance under that target density. Then we show that an auto-encoder with a contractive penalty captures estimators of these local moments in its reconstruction function and its Jacobian. A contribution of this work is thus a novel alternative to maximum-likelihood density estimation, which we call local moment matching. It also justifies a recently proposed sampling algorithm for the Contractive Auto-Encoder and extends it to the Denoising Auto-Encoder. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Algorithms to Disentangle Factors of Variation <s> It has been hypothesized, and supported with experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce Markov chains that mix faster between modes. Consequently, mixing between modes would be more efficient at higher levels of representation. To better understand this, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing between modes and interpolating between samples. <s> BIB003
The goal of building invariant features is to remove sensitivity of the representation to directions of variance in the data that are uninformative to the task at hand. However, it is often the case that the goal of feature extraction is the disentangling or separation of many distinct but informative factors in the data, for example, in a video of people: subject identity, action performed, subject pose relative to the camera, and so on. In this situation, the methods of generating invariant features, such as feature pooling, may be inadequate. The process of building invariant features can be seen as consisting of two steps. First, low-level features are recovered that account for the data. Second, subsets of these low-level features are pooled together to form higher level invariant features, exemplified by the pooling and subsampling layers of convolutional neural networks. The invariant representation formed by the pooling features offers an incomplete window on the data as the detailed representation of the lower level features is abstracted away in the pooling procedure. While we would like higher level features to be more abstract and exhibit greater invariance, we have little control over what information is lost through pooling. What we really would like is for a particular feature set to be invariant to the irrelevant features and disentangle the relevant features. Unfortunately, it is often difficult to determine a priori which set of features will ultimately be relevant to the task at hand. An interesting approach to taking advantage of some of the factors of variation known to exist in the data is the transforming autoencoder BIB001 : Instead of a scalar pattern detector (e.g., corresponding to the probability of the presence of a particular form in the input), one can think of the features as organized in groups that include both a pattern detector and pose parameters that specify attributes of the detected pattern. In BIB001 , what is assumed a priori is that pairs of examples (or consecutive ones) are observed with an associated value for the corresponding change in the pose parameters. For example, an animal that controls its eyes knows what changes to its ocular motor system were applied when going from one image on its retina to the next. In that work, it is also assumed that the pose changes are the same for all the pattern detectors, and this makes sense for global changes such as image translation and camera geometry changes. Instead, we would like to discover the pose parameters and attributes that should be associated with each feature detector without having to specify ahead of time what they should be, force them to be the same for all features, and having to necessarily observe the changes in all of the pose parameters or attributes. The approach taken recently in the MTC, discussed in Section 8.3, is interesting in this respect. Without any supervision or prior knowledge, it finds prominent local factors of variation (tangent vectors to the manifold, extracted from a CAE, interpreted as locally valid input "deformations"). Higher level features are subsequently encouraged to be invariant to these factors of variation so that they must depend on other characteristics. In a sense, this approach is disentangling valid local deformations along the data manifold from other, more drastic changes associated to other factors of variation such as those that affect class identity. BIB002 One solution to the problem of information loss that would fit within the feature-pooling paradigm is to consider many overlapping pools of features based on the same low-level feature set. Such a structure would have the potential to learn a redundant set of invariant features that may not cause significant loss of information. However, it is not obvious what learning principle could be applied that can ensure that the features are invariant while maintaining as much information as possible. While a DBN or a DBM (as discussed in Sections 4 and 10.2, respectively) with two hidden layers would, in principle, be able to preserve information into the "pooling" second hidden layer, there is no guarantee that the second layer features are more invariant than the "low-level" first layer features. However, there is some empirical evidence that the second layer of the DBN tends to display more invariance than the first layer  . A more principled approach, from the perspective of ensuring a more robust compact feature representation, can be conceived by reconsidering the disentangling of features through the lens of its generative equivalent-feature composition. Since many unsupervised learning algorithms have a generative interpretation (or a way to reconstruct inputs from their high-level representation), the generative perspective can provide insight into how to think about disentangling factors. The majority of the models currently used to construct invariant features have the interpretation that their low-level features linearly combine to construct the data. BIB003 This is a fairly rudimentary form of feature composition with significant limitations. For example, it is not possible to linearly combine a feature with a generic transformation (such as translation) to generate a transformed version of the feature. Nor can we even consider a generic color feature being linearly combined with a grayscale stimulus pattern to generate a colored pattern. It would seem that if we are to take the notion of disentangling seriously, we require a richer interaction of features than that offered by simple linear combinations.