Representation Learning: A Review and New Perspectives <s> PCA <s> Abstract The separation of independent sources from an array of sensors is a classical but difficult problem in signal processing. Based on some biological observations, an adaptive algorithm is proposed to separate simultaneously all the unknown independent sources. The adaptive rule, which constitutes an independence test using non-linear functions, is the main original point of this blind identification procedure. Moreover, a new concept, that of INdependent Components Analysis (INCA), more powerful than the classical Principal Components Analysis (in decision tasks) emerges from this work. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> PCA <s> In ordinary independent component analysis, the components are assumed to be completely independent, and they do not necessarily have any meaningful order relationships. In practice, however, the estimated "independent" components are often not at all independent. We propose that this residual dependence structure could be used to define a topographic order for the components. In particular, a distance between two components could be defined using their higher-order correlations, and this distance could be used to create a topographic representation. Thus, we obtain a linear decomposition into approximately independent components, where the dependence of two components is approximated by the proximity of the components in the topographic representation. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> PCA <s> Convolutional neural networks (CNNs) have been successfully applied to many tasks such as digit and object recognition. Using convolutional (tied) weights significantly reduces the number of parameters that have to be learned, and also allows translational invariance to be hard-coded into the architecture. In this paper, we consider the problem of learning invariances, rather than relying on hard-coding. We propose tiled convolution neural networks (Tiled CNNs), which use a regular "tiled" pattern of tied weights that does not require that adjacent hidden units share identical weights, but instead requires only that hidden units k steps away from each other to have tied weights. By pooling over neighboring units, this architecture is able to learn complex invariances (such as scale and rotational invariance) beyond translational invariance. Further, it also enjoys much of CNNs' advantage of having a relatively small number of learned parameters (such as ease of learning and greater scalability). We provide an efficient learning algorithm for Tiled CNNs based on Topographic ICA, and show that learning complex invariant features allows us to achieve highly competitive results for both the NORB and CIFAR-10 datasets. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> PCA <s> Previous work on action recognition has focused on adapting hand-designed local features, such as SIFT or HOG, from static images to the video domain. In this paper, we propose using unsupervised feature learning as a way to learn features directly from video data. More specifically, we present an extension of the Independent Subspace Analysis algorithm to learn invariant spatio-temporal features from unlabeled video data. We discovered that, despite its simplicity, this method performs surprisingly well when combined with deep learning techniques such as stacking and convolution to learn hierarchical representations. By replacing hand-designed features with our learned features, we achieve classification results superior to all previous published results on the Hollywood2, UCF, KTH and YouTube action recognition datasets. On the challenging Hollywood2 and YouTube action datasets we obtain 53.3% and 75.8% respectively, which are approximately 5% better than the current best published results. Further benefits of this method, such as the ease of training and the efficiency of training and prediction, will also be discussed. You can download our code and learned spatio-temporal features here: http://ai.stanford.edu/∼wzou/ <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> PCA <s> Stacked denoising autoencoders (SDAs) have been successfully used to learn new representations for domain adaptation. Recently, they have attained record accuracy on standard benchmark tasks of sentiment analysis across different text domains. SDAs learn robust data representations by reconstruction, recovering original features from data that are artificially corrupted with noise. In this paper, we propose marginalized SDA (mSDA) that addresses two crucial limitations of SDAs: high computational cost and lack of scalability to high-dimensional features. In contrast to SDAs, our approach of mSDA marginalizes noise and thus does not require stochastic gradient descent or other optimization algorithms to learn parameters--in fact, they are computed in closed-form. Consequently, mSDA, which can be implemented in only 20 lines of MATLAB™, significantly speeds up SDAs by two orders of magnitude. Furthermore, the representations learnt by mSDA are as effective as the traditional SDAs, attaining almost identical accuracies in benchmark tasks. <s> BIB005
We will use probably the oldest feature extraction algorithm, PCA, to illustrate the probabilistic, autoencoder, and manifold views of representation learning. PCA learns a linear transformation h ¼ fðxÞ where the columns of d x Â d h matrix W form an orthogonal basis for the d h orthogonal directions of greatest variance in the training data. The result is d h features (the components of representation h) that are decorrelated. The three interpretations of PCA are the following: 1) It is related to probabilistic models (Section 6) such as probabilistic PCA, factor analysis, and the traditional multivariate Gaussian distribution (the leading eigenvectors of the covariance matrix are the principal components); 2) the representation it learns is essentially the same as that learned by a basic linear autoencoder (Section 7.2); and 3) it can be viewed as a simple linear form of linear manifold learning (Section 8), i.e., characterizing a lower dimensional region in input space near which the data density is peaked. Thus, PCA may be in the back of the reader's mind as a common thread relating these various viewpoints. Unfortunately, the expressive power of linear features is very limited: They cannot be stacked to form deeper, more abstract representations since the composition of linear operations yields another linear operation. Here, we focus on recent algorithms that have been developed to extract nonlinear features, which can be stacked in the construction of deep networks, although some authors simply insert a nonlinearity between learned single-layer linear projections BIB004 , BIB005 . Another rich family of feature extraction techniques that this review does not cover in any detail due to space constraints is independent component analysis or ICA BIB001 ,  . Instead, we refer the reader to [101] ,  . Note that while in the simplest case (complete, noise free) ICA yields linear features, in the more general case it can be equated with a linear generative model with non-Gaussian independent latent variables, similar to sparse coding (Section 6.1.1), which result in nonlinear features. Therefore, ICA and its variants like independent and topographic ICA BIB002 can and have been used to build deep networks BIB003 , BIB004 (see Section 11.2). The notion of obtaining independent components also appears similar to our stated goal of disentangling underlying explanatory factors through deep networks. However, for complex real-world distributions, it is doubtful that the relationship between truly independent underlying factors and the observed high-dimensional data can be adequately characterized by a linear transformation.