Representation Learning: A Review and New Perspectives <s> DIRECTLY LEARNING A PARAMETRIC MAP FROM INPUT TO REPRESENTATION <s> Invariant features of temporally varying signals are useful for analysis and classification. Slow feature analysis (SFA) is a new method for learning invariant or slowly varying features from a vectorial input signal. It is based on a nonlinear expansion of the input signal and application of principal component analysis to this expanded signal and its time derivative. It is guaranteed to find the optimal solution within a family of functions directly and can learn to extract a large number of decorrelated features, which are ordered by their degree of invariance. SFA can be applied hierarchically to process high-dimensional input signals and extract complex features. SFA is applied first to complex cell tuning properties based on simple cell output, including disparity and motion. Then more complicated input-output functions are learned by repeated application of SFA. Finally, a hierarchical network of SFA modules is presented as a simple model of the visual system. The same unstructured network can learn translation, size, rotation, contrast, or, to a lesser degree, illumination invariance for one-dimensional objects, depending on only the training stimulus. Surprisingly, only a few training objects suffice to achieve good generalization to new objects. The generated representation is suitable for object recognition. Performance degrades if the network is trained to learn multiple invariances simultaneously. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> DIRECTLY LEARNING A PARAMETRIC MAP FROM INPUT TO REPRESENTATION <s> We show how nonlinear embedding algorithms popular for use with shallow semi-supervised learning techniques such as kernel methods can be applied to deep multilayer architectures, either as a regularizer at the output layer, or on each layer of the architecture. This provides a simple alternative to existing approaches to deep learning whilst yielding competitive error rates compared to those methods, and existing shallow semi-supervised techniques. <s> BIB002
Within the framework of probabilistic models adopted in Section 6, the learned representation is always associated with latent variables, specifically with their posterior distribution given an observed input x. Unfortunately, this posterior distribution tends to become very complicated and intractable if the model has more than a couple of interconnected layers, whether in the directed or undirected graphical model frameworks. It then becomes necessary to resort to sampling or approximate inference techniques, and to pay the associated computational and approximation error price. If the true posterior has a large number of modes that matter, then current inference techniques may face an unsurmountable challenge or endure a potentially serious approximation. This is in addition to the difficulties raised by the intractable partition function in undirected graphical models. Moreover, a posterior distribution over latent variables is not yet a simple usable feature vector that can, for example, be fed to a classifier. So, actual feature values are typically derived from that distribution, taking the latent variable's expectation (as is typically done with RBMs), their marginal probability, or finding their most likely value (as in sparse coding). If we are to extract stable deterministic numerical feature values in the end anyway, an alternative (apparently) nonprobabilistic feature learning paradigm that focuses on carrying out this part of the computation very efficiently is that of autoencoders and other directly parameterized feature or representation functions. The commonality between these methods is that they learn a direct encoding, i.e., a parametric map from inputs to their representation. Regularized autoencoders, discussed next, also involve learning a decoding function that maps back from representation to input space. Sections 8.1 and 11.3 discuss direct encoding methods that do not require a decoder, such as semi-supervised embedding BIB002 and slow feature analysis BIB001 .