Representation Learning: A Review and New Perspectives <s> Explaining Away <s> THE receptive fields of simple cells in mammalian primary visual cortex can be characterized as being spatially localized, oriented1–4 and bandpass (selective to structure at different spatial scales), comparable to the basis functions of wavelet transforms5,6. One approach to understanding such response properties of visual neurons has been to consider their relationship to the statistical structure of natural images in terms of efficient coding7–12. Along these lines, a number of studies have attempted to train unsupervised learning algorithms on natural images in the hope of developing receptive fields with similar properties13–18, but none has succeeded in producing a full set that spans the image space and contains all three of the above properties. Here we investigate the proposal8,12 that a coding strategy that maximizes sparseness is sufficient to account for these properties. We show that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex. The resulting sparse image code provides a more efficient representation for later stages of processing because it possesses a higher degree of statistical independence among its outputs. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> When a parameter space has a certain underlying structure, the ordinary gradient of a function does not represent its steepest direction, but the natural gradient does. Information geometry is used for calculating the natural gradients in the parameter space of perceptrons, the space of matrices (for blind source separation), and the space of linear dynamical systems (for blind source deconvolution). The dynamical behavior of natural gradient online learning is analyzed and is proved to be Fisher efficient, implying that it has asymptotically the same performance as the optimal batch estimation of parameters. This suggests that the plateau phenomenon, which appears in the backpropagation learning algorithm of multilayer perceptrons, might disappear or might not be so serious when the natural gradient is used. An adaptive method of updating the learning rate is proposed and analyzed. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> Although the basic sparse coding model has been quite successful at explaining the receptive fields of simple cells in V1, it ignores an important constrain: perception task. We put forward a novel sparse coding model, called task-oriented sparse coding (TOSC) model, combining the discriminability constrain supervised by classification task, besides the sparseness criteria. Simulation experiments are performed using real images including class of scene and class of building. The results show that TOSC can organize some significant receptive fields with distinct topological structure which will favor the classification task. Moreover, the coefficients of TOSC notablely improve the classification accuracy, from the 53.5% of pixel-based model to 86.7%, in the case of none distinct damage on the performance of reconstruction error and sparseness. TOSC model, complementing the feedback sparse coding model, is more consistent with biological mechanism, and shows good potential in the feature extraction for pattern classification. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> We present a new machine learning framework called "self-taught learning" for using unlabeled data in supervised classification tasks. We do not assume that the unlabeled data follows the same class labels or generative distribution as the labeled data. Thus, we would like to use a large number of unlabeled images (or audio samples, or text documents) randomly downloaded from the Internet to improve performance on a given image (or audio, or text) classification task. Such unlabeled data is significantly easier to obtain than in typical semi-supervised or transfer learning settings, making self-taught learning widely applicable to many practical learning problems. We describe an approach to self-taught learning that uses sparse coding to construct higher-level features using the unlabeled data. These features form a succinct input representation and significantly improve classification performance. When using an SVM for classification, we further show how a Fisher kernel can be learned for this representation. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> Prior work has shown that features which appear to be biologically plausible as well as empirically useful can be found by sparse coding with a prior such as a laplacian (L1) that promotes sparsity. We show how smoother priors can preserve the benefits of these sparse priors while adding stability to the Maximum A-Posteriori (MAP) estimate that makes it more useful for prediction problems. Additionally, we show how to calculate the derivative of the MAP estimate efficiently with implicit differentiation. One prior that can be differentiated this way is KL-regularization. We demonstrate its effectiveness on a wide variety of applications, and find that online optimization of the parameters of the KL-regularized model can significantly improve prediction performance. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> Can machine learning deliver AI? Theoretical results, inspiration from the brain and cognition, as well as machine learning experiments suggest that in order to learn the kind of complicated functions that can represent high-level abstractions (e.g. in vision, language, and other AI-level tasks), one would need deep architectures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers, graphical models with many levels of latent variables, or in complicated propositional formulae re-using many sub-formulae. Each level of the architecture represents features at a different level of abstraction, defined as a composition of lower-level features. Searching the parameter space of deep architectures is a difficult task, but new algorithms have been discovered and a new sub-area has emerged in the machine learning community since 2006, following these discoveries. Learning algorithms such as those for Deep Belief Networks and other related unsupervised learning algorithms have recently been proposed to train deep architectures, yielding exciting results and beating the state-of-the-art in certain areas. Learning Deep Architectures for AI discusses the motivations for and principles of learning algorithms for deep architectures. By analyzing and comparing recent results with different learning algorithms for deep architectures, explanations for their success are proposed and discussed, highlighting challenges and suggesting avenues for future explorations in this area. <s> BIB006 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> Several recently-proposed architectures for high-performance object recognition are composed of two main stages: a feature extraction stage that extracts locally-invariant feature vectors from regularly spaced image patches, and a somewhat generic supervised classifier. The first stage is often composed of three main modules: (1) a bank of filters (often oriented edge detectors); (2) a non-linear transform, such as a point-wise squashing functions, quantization, or normalization; (3) a spatial pooling operation which combines the outputs of similar filters over neighboring regions. We propose a method that automatically learns such feature extractors in an unsupervised fashion by simultaneously learning the filters and the pooling units that combine multiple filter outputs together. The method automatically generates topographic maps of similar filters that extract features of orientations, scales, and positions. These similar filters are pooled together, producing locally-invariant outputs. The learned feature descriptors give comparable results as SIFT on image recognition tasks for which SIFT is well suited, and better results than SIFT on tasks for which SIFT is less well suited. <s> BIB007 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> We consider the empirical risk minimization problem for linear supervised learning, with regularization by structured sparsity-inducing norms. These are defined as sums of Euclidean norms on certain subsets of variables, extending the usual $\ell_1$-norm and the group $\ell_1$-norm by allowing the subsets to overlap. This leads to a specific set of allowed nonzero patterns for the solutions of such problems. We first explore the relationship between the groups defining the norm and the resulting nonzero patterns, providing both forward and backward algorithms to go back and forth from groups to patterns. This allows the design of norms adapted to specific prior knowledge expressed in terms of nonzero patterns. We also present an efficient active set algorithm, and analyze the consistency of variable selection for least-squares linear regression in low and high-dimensional settings. <s> BIB008 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> Adaptive sparse coding methods learn a possibly overcomplete set of basis functions, such that natural image patches can be reconstructed by linearly combining a small subset of these bases. The applicability of these methods to visual object recognition tasks has been limited because of the prohibitive cost of the optimization algorithms required to compute the sparse representation. In this work we propose a simple and efficient algorithm to learn basis functions. After training, this model also provides a fast and smooth approximator to the optimal representation, achieving even better accuracy than exact sparse coding algorithms on visual object recognition tasks. <s> BIB009 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> Penalized least squares regression is often used for signal denoising and inverse problems, and is commonly interpreted in a Bayesian framework as a Maximum a posteriori (MAP) estimator, the penalty function being the negative logarithm of the prior. For example, the widely used quadratic program (with an l1 penalty) associated to the LASSO/basis pursuit denoising is very often considered as MAP estimation under a Laplacian prior in the context of additive white Gaussian noise (AWGN) reduction. This paper highlights the fact that, while this is one possible Bayesian interpretation, there can be other equally acceptable Bayesian interpretations. Therefore, solving a penalized least squares regression problem with penalty φ(x) need not be interpreted as assuming a prior C·exp(-φ(x)) and using the MAP estimator. In particular, it is shown that for any prior PX, the minimum mean-square error (MMSE) estimator is the solution of a penalized least square problem with some penalty φ(x) , which can be interpreted as the MAP estimator with the prior C·exp(-φ(x)). Vice versa, for certain penalties φ(x), the solution of the penalized least squares problem is indeed the MMSE estimator, with a certain prior PX . In general dPX(x) ≠ C·exp(-φ(x))dx. <s> BIB010 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> While vector quantization (VQ) has been applied widely to generate features for visual recognition problems, much recent work has focused on more powerful methods. In particular, sparse coding has emerged as a strong alternative to traditional VQ approaches and has been shown to achieve consistently higher performance on benchmark datasets. Both approaches can be split into a training phase, where the system learns a dictionary of basis functions, and an encoding phase, where the dictionary is used to extract features from new inputs. In this work, we investigate the reasons for the success of sparse coding over VQ by decoupling these phases, allowing us to separate out the contributions of training and encoding in a controlled way. Through extensive experiments on CIFAR, NORB and Caltech 101 datasets, we compare several training and encoding schemes, including sparse coding and a form of VQ with a soft threshold activation function. Our results show not only that we can use fast VQ algorithms for training, but that we can just as well use randomly chosen exemplars from the training set. Rather than spend resources on training, we find it is more important to choose a good encoder—which can often be a simple feed forward non-linearity. Our results include state-of-the-art performance on both CIFAR and NORB. <s> BIB011 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> We present a method for learning image representations using a two-layer sparse coding scheme at the pixel level. The first layer encodes local patches of an image. After pooling within local regions, the first layer codes are then passed to the second layer, which jointly encodes signals from the region. Unlike traditional sparse coding methods that encode local patches independently, this approach accounts for high-order dependency among patterns in a local image neighborhood. We develop algorithms for data encoding and codebook learning, and show in experiments that the method leads to more invariant and discriminative image representations. The algorithm gives excellent results for hand-written digit recognition on MNIST and object recognition on the Caltech101 benchmark. This marks the first time that such accuracies have been achieved using automatically learned features from the pixel level, rather than using hand-designed descriptors. <s> BIB012 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. While naturally cast as a combinatorial optimization problem, variable or feature selection admits a convex relaxation through the regularization by the $\ell_1$-norm. In this paper, we consider situations where we are not only interested in sparsity, but where some structural prior knowledge is available as well. We show that the $\ell_1$-norm can then be extended to structured norms built on either disjoint or overlapping groups of variables, leading to a flexible framework that can deal with various structures. We present applications to unsupervised learning, for structured sparse principal component analysis and hierarchical dictionary learning, and to supervised learning in the context of non-linear variable selection. <s> BIB013 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> This work describes a conceptually simple method for structured sparse coding and dictionary design. Supposing a dictionary with K atoms, we introduce a structure as a set of penalties or interactions between every pair of atoms. We describe modifications of standard sparse coding algorithms for inference in this setting, and describe experiments showing that these algorithms are efficient. We show that interesting dictionaries can be learned for interactions that encode tree structures or locally connected structures. Finally, we show that our framework allows us to learn the values of the interactions from the data, rather than having them pre-specified. <s> BIB014 </s> Representation Learning: A Review and New Perspectives <s> Explaining Away <s> We consider the problem of using a factor model we call {\em spike-and-slab sparse coding} (S3C) to learn features for a classification task. The S3C model resembles both the spike-and-slab RBM and sparse coding. Since exact inference in this model is intractable, we derive a structured variational inference procedure and employ a variational EM training algorithm. Prior work on approximate inference for this model has not prioritized the ability to exploit parallel architectures and scale to enormous problem sizes. We present an inference procedure appropriate for use with GPUs which allows us to dramatically increase both the training set size and the amount of latent factors. ::: We demonstrate that this approach improves upon the supervised learning capabilities of both sparse coding and the ssRBM on the CIFAR-10 dataset. We evaluate our approach's potential for semi-supervised learning on subsets of CIFAR-10. We demonstrate state-of-the art self-taught learning performance on the STL-10 dataset and use our method to win the NIPS 2011 Workshop on Challenges In Learning Hierarchical Models' Transfer Learning Challenge. <s> BIB015
Directed models often lead to one important property: explaining away, i.e., a priori independent causes of an event can become nonindependent given the observation of the event. Latent factor models can generally be interpreted as latent cause models, where the h activations cause the observed x. This renders the a priori independent h to be nonindependent. As a consequence, recovering the posterior distribution of h, pðh j xÞ (which we use as a basis for feature representation), is often computationally challenging and can be entirely intractable, especially when h is discrete. A classic example that illustrates the phenomenon is to imagine you are on vacation away from home and you receive a phone call from the security system company telling you that the alarm has been activated. You begin worrying your home has been burglarized, but then you hear on the radio that a minor earthquake has been reported in the area of your home. If you happen to know from prior experience that earthquakes sometimes cause your home alarm system to activate, then suddenly you relax, confident that your home has very likely not been burglarized. The example illustrates how the alarm activation rendered two otherwise entirely independent causes, burglarized and earthquake, to become dependent-in this case, the dependency is one of mutual exclusivity. Since both burglarized and earthquake are very rare events and both can cause alarm activation, the observation of one explains away the other. Despite the computational obstacles we face when attempting to recover the posterior over h, explaining away promises to provide a parsimonious pðh j xÞ, which can be an extremely useful characteristic of a feature encoding scheme. If one thinks of a representation as being composed of various feature detectors and estimated attributes of the observed input, it is useful to allow the different features to compete and collaborate with each other to explain the input. This is naturally achieved with directed graphical models, but can also be achieved with undirected models (see Section 6.2) such as Boltzmann machines if there are lateral connections between the corresponding units or corresponding interaction terms in the energy function that defines the probability model. Probabilistic interpretation of PCA. PCA can be given a natural probabilistic interpretation  ,  as factor analysis: where x 2 IR d x , h 2 IR d h , N ðv; ; AEÞ is the multivariate normal density of v with mean and covariance AE, and columns of W span the same space as leading d h principal components, but are not constrained to be orthonormal. Sparse coding. Like PCA, sparse coding has both a probabilistic and nonprobabilistic interpretation. Sparse coding also relates a latent representation h (either a vector of random variables or a feature vector, depending on the interpretation) to the data x through a linear mapping W , which we refer to as the dictionary. The difference between sparse coding and PCA is that sparse coding includes a penalty to ensure a sparse activation of h is used to encode each input x. From a nonprobabilistic perspective, sparse coding can be seen as recovering the code or feature vector associated with a new input x via Learning the dictionary W can be accomplished by optimizing the following training criterion with respect to W : where x ðtÞ is the tth example and h ÃðtÞ is the corresponding sparse code determined by BIB002 . W is usually constrained to have unit-norm columns (because one can arbitrarily exchange scaling of column i with scaling of h ðtÞ i , such a constraint is necessary for the L1 penalty to have any effect). The probabilistic interpretation of sparse coding differs from that of PCA in that instead of a Gaussian prior on the latent random variable h, we use a sparsity inducing Laplace prior (corresponding to an L1 penalty): In the case of sparse coding, because we will seek a sparse representation (i.e., one with many features set to exactly zero), we will be interested in recovering the maximum a posteriori value (MAP value of h, i.e., h Ã ¼ argmax h pðh j xÞ rather than its expected value IE½h j x). Under this interpretation, dictionary learning proceeds as maximizing the likelihood of the data given these MAP values of h Ã : argmax W Q t pðx ðtÞ j h ÃðtÞ Þ subject to the norm constraint on W . Note that this parameter learning scheme, subject to the MAP values of the latent h, is not standard practice in the probabilistic graphical model literature. Typically, the likelihood of the data pðxÞ ¼ P h pðx j hÞpðhÞ is maximized directly. In the presence of latent variables, expectation maximization is employed, where the parameters are optimized with respect to the marginal likelihood, i.e., summing or integrating the joint log likelihood over all values of the latent variables under their posterior P ðh j xÞ, rather than considering only the single MAP value of h. The theoretical properties of this form of parameter learning are not yet well understood but seem to work well in practice (e.g., k-means versus Gaussian mixture models and Viterbi training for HMMs). Note also that the interpretation of sparse coding as a MAP estimation can be questioned BIB010 because, even though the interpretation of the L1 penalty as a log prior is a possible interpretation, there can be other Bayesian interpretations compatible with the training criterion. Sparse coding is an excellent example of the power of explaining away. Even with a very overcomplete dictionary, BIB006 the MAP inference process used in sparse coding to find h Ã can pick out the most appropriate bases and zero the others, despite them having a high degree of correlation with the input. This property arises naturally in directed graphical models such as sparse coding and is entirely due to the explaining away effect. It is not seen in commonly used undirected probabilistic models such as the RBM, nor is it seen in parametric feature encoding methods such as autoencoders. The tradeoff is that compared to methods such as RBMs and autoencoders, inference in sparse coding involves an extra inner loop of optimization to find h Ã with a corresponding increase in the computational cost of feature extraction. Compared to autoencoders and RBMs, the code in sparse coding is a free variable for each example, and in that sense, the implicit encoder is nonparametric. One might expect that the parsimony of the sparse coding representation and its explaining away effect would be advantageous and indeed it seems to be the case. Coates and Ng BIB011 demonstrated on the CIFAR-10 object classification task  with a patch-base feature extraction pipeline that in the regime with few (<1;000) labeled training examples per class, the sparse coding representation significantly outperformed other highly competitive encoding schemes. Possibly because of these properties and because of the very computationally efficient algorithms that have been proposed for it (in comparison with the general case of inference in the presence of explaining away), sparse coding enjoys considerable popularity as a feature learning and encoding paradigm. There are numerous examples of its successful application as a feature representation scheme, including natural image modeling BIB004 , BIB009 , BIB011 , BIB012 , audio classification BIB003 , NLP BIB005 , as well as being a very successful model of the early visual cortex BIB001 . Sparsity criteria can also be generalized successfully to yield groups of features that prefer to all be zero, but if one or a few of them are active, then the penalty for activating others in the group is small. Different group sparsity patterns can incorporate different forms of prior knowledge BIB007 , BIB008 , BIB013 , BIB014 . S3C. S3C is one example of a promising variation on sparse coding for feature learning BIB015 . The S3C model possesses a set of latent binary spike variables together with a set of latent real-valued slab variables. The activation of the spike variables dictates the sparsity pattern. S3C has been applied to the CIFAR-10 and CIFAR-100 object classification tasks  , and shows the same pattern as sparse coding of superior performance in the regime of relatively few (< 1;000) labeled examples per class BIB015 . In fact, in both the CIFAR-100 dataset (with 500 examples per class) and the CIFAR-10 dataset (when the number of examples is reduced to a similar range), the S3C representation actually outperforms sparse coding representations. This advantage was revealed clearly with S3C winning the NIPS '11 Transfer Learning Challenge BIB015 .