Representation Learning: A Review and New Perspectives <s> Generalizations of the RBM to Real-Valued Data <s> We propose a model for natural images in which the probability of an image is proportional to the product of the probabilities of some filter outputs. We encourage the system to find sparse features by using a Student-t distribution to model each filter output. If the t-distribution is used to model the combined outputs of sets of neurally adjacent filters, the system learns a topographic map in which the orientation, spatial frequency and location of the filters change smoothly across the map. Even though maximum likelihood learning is intractable in our model, the product form allows a relatively efficient learning procedure that works well even for highly overcomplete sets of filters. Once the model has been learned it can be used as a prior to derive the "iterated Wiener filter" for the purpose of denoising images. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Generalizations of the RBM to Real-Valued Data <s> THE receptive fields of simple cells in mammalian primary visual cortex can be characterized as being spatially localized, oriented1–4 and bandpass (selective to structure at different spatial scales), comparable to the basis functions of wavelet transforms5,6. One approach to understanding such response properties of visual neurons has been to consider their relationship to the statistical structure of natural images in terms of efficient coding7–12. Along these lines, a number of studies have attempted to train unsupervised learning algorithms on natural images in the hope of developing receptive fields with similar properties13–18, but none has succeeded in producing a full set that spans the image space and contains all three of the above properties. Here we investigate the proposal8,12 that a coding strategy that maximizes sparseness is sufficient to account for these properties. We show that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex. The resulting sparse image code provides a more efficient representation for later stages of processing because it possesses a higher degree of statistical independence among its outputs. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Generalizations of the RBM to Real-Valued Data <s> Although the basic sparse coding model has been quite successful at explaining the receptive fields of simple cells in V1, it ignores an important constrain: perception task. We put forward a novel sparse coding model, called task-oriented sparse coding (TOSC) model, combining the discriminability constrain supervised by classification task, besides the sparseness criteria. Simulation experiments are performed using real images including class of scene and class of building. The results show that TOSC can organize some significant receptive fields with distinct topological structure which will favor the classification task. Moreover, the coefficients of TOSC notablely improve the classification accuracy, from the 53.5% of pixel-based model to 86.7%, in the case of none distinct damage on the performance of reconstruction error and sparseness. TOSC model, complementing the feedback sparse coding model, is more consistent with biological mechanism, and shows good potential in the feature extraction for pattern classification. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Generalizations of the RBM to Real-Valued Data <s> Learning a generative model of natural images is a useful way of extracting features that capture interesting regularities. Previous work on learning such models has focused on methods in which the latent features are used to determine the mean and variance of each pixel independently, or on methods in which the hidden units determine the covariance matrix of a zero-mean Gaussian distribution. In this work, we propose a probabilistic model that combines these two approaches into a single framework. We represent each image using one set of binary latent features that model the image-specific covariance and a separate set that model the mean. We show that this approach provides a probabilistic framework for the widely used simple-cell complex-cell architecture, it produces very realistic samples of natural images and it extracts features that yield state-of-the-art recognition accuracy on the challenging CIFAR 10 dataset. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> Generalizations of the RBM to Real-Valued Data <s> Deep belief nets have been successful in modeling handwritten characters, but it has proved more difficult to apply them to real images. The problem lies in the restricted Boltzmann machine (RBM) which is used as a module for learning deep belief nets one layer at a time. The Gaussian-Binary RBMs that have been used to model real-valued data are not a good way to model the covariance structure of natural images. We propose a factored 3-way RBM that uses the states of its hidden units to represent abnormalities in the local covariance structure of an image. This provides a probabilistic framework for the widely used simple/complex cell architecture. Our model learns binary features that work very well for object recognition on the “tiny images” data set. Even better features are obtained by then using standard binary RBM’s to learn a deeper model. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> Generalizations of the RBM to Real-Valued Data <s> Straightforward application of Deep Belief Nets (DBNs) to acoustic modeling produces a rich distributed representation of speech data that is useful for recognition and yields impressive results on the speaker-independent TIMIT phone recognition task. However, the first-layer Gaussian-Bernoulli Restricted Boltzmann Machine (GRBM) has an important limitation, shared with mixtures of diagonal-covariance Gaussians: GRBMs treat different components of the acoustic input vector as conditionally independent given the hidden state. The mean-covariance restricted Boltzmann machine (mcRBM), first introduced for modeling natural images, is a much more representationally efficient and powerful way of modeling the covariance structure of speech data. Every configuration of the precision units of the mcRBM specifies a different precision matrix for the conditional distribution over the acoustic space. In this work, we use the mcRBM to learn features of speech data that serve as input into a standard DBN. The mcRBM features combined with DBNs allow us to achieve a phone error rate of 20.5%, which is superior to all published results on speaker-independent TIMIT to date. <s> BIB006 </s> Representation Learning: A Review and New Perspectives <s> Generalizations of the RBM to Real-Valued Data <s> Probabilistic models of natural images are usually evaluated by measuring performance on rather indirect tasks, such as denoising and inpainting. A more direct way to evaluate a generative model is to draw samples from it and to check whether statistical properties of the samples match the statistics of natural images. This method is seldom used with high-resolution images, because current models produce samples that are very different from natural images, as assessed by even simple visual inspection. We investigate the reasons for this failure and we show that by augmenting existing models so that there are two sets of latent variables, one set modelling pixel intensities and the other set modelling image-specific pixel covariances, we are able to generate high-resolution images that look much more realistic than before. The overall model can be interpreted as a gated MRF where both pair-wise dependencies and mean intensities of pixels are modulated by the states of latent variables. Finally, we confirm that if we disallow weight-sharing between receptive fields that overlap each other, the gated MRF learns more efficient internal representations, as demonstrated in several recognition tasks. <s> BIB007 </s> Representation Learning: A Review and New Perspectives <s> Generalizations of the RBM to Real-Valued Data <s> We introduce the spike and slab Restricted Boltzmann Machine, characterized by having both a real-valued vector, the slab, and a binary variable, the spike, associated with each unit in the hidden layer. The model possesses some practical properties such as being amenable to Block Gibbs sampling as well as being capable of generating similar latent representations of the data to the recently introduced mean and covariance Restricted Boltzmann Machine. We illustrate how the spike and slab Restricted Boltzmann Machine achieves competitive performance on the CIFAR-10 object recognition task. <s> BIB008 </s> Representation Learning: A Review and New Perspectives <s> Generalizations of the RBM to Real-Valued Data <s> We assess the generative power of the mPoTmodel of [10] with tiled-convolutional weight sharing as a model for visual textures by specifically training on this task, evaluating model performance on texture synthesis and inpainting tasks using quantitative metrics. We also analyze the relative importance of the mean and covariance parts of the mPoT model by comparing its performance to those of its subcomponents, tiled-convolutional versions of the PoT/FoE and Gaussian-Bernoulli restricted Boltzmann machine (GB-RBM). Our results suggest that while state-of-the-art or better performance can be achieved using the mPoT, similar performance can be achieved with the mean-only model. We then develop a model for multiple textures based on the GB-RBM, using a shared set of weights but texturespecific hidden unit biases. We show comparable performance of the multiple texture model to individually trained texture models. <s> BIB009
Important progress has been made in the last few years in defining generalizations of the RBM that better capture realvalued data, in particular real-valued image data, by better modeling the conditional covariance of the input pixels. The standard RBM, as discussed above, is defined with both binary visible variables v 2 f0; 1g and binary latent variables h 2 f0; 1g. The tractability of inference and learning in the RBM has inspired many authors to extend it, via modifications of its energy function, to model other kinds of data distributions. In particular, there have been multiple attempts to develop RBM-type models of real-valued data, where x 2 IR dx . The most straightforward approach to modeling real-valued observations within the RBM framework is the so-called Gaussian RBM (GRBM), where the only change in the RBM energy function is to the visible units biases, by adding a bias term that is quadratic in the visible units x. While it probably remains the most popular way to model real-valued data within the RBM framework, Ranzato and Hinton BIB004 suggest that the GRBM has proven to be a somewhat unsatisfactory model of natural images. The trained features typically do not represent sharp edges that occur at object boundaries and lead to latent representations that are not particularly useful features for classification tasks. Ranzato and Hinton BIB004 argue that the failure of the GRBM to adequately capture the statistical structure of natural images stems from the exclusive use of the model capacity to capture the conditional mean at the expense of the conditional covariance. Natural images, they argue, are chiefly characterized by the covariance of the pixel values, not by their absolute values. This point is supported by the common use of preprocessing methods that standardize the global scaling of the pixel values across images in a dataset or across the pixel values within each image. These kinds of concerns about the ability of the GRBM to model natural image data has led to the development of alternative RBM-based models that each attempt to take on this objective of better modeling nondiagonal conditional covariances. Ranzato and Hinton BIB004 introduced the mean and covariance RBM (mcRBM). Like the GRBM, the mcRBM is a two-layer Boltzmann machine that explicitly models the visible units as Gaussian distributed quantities. However, unlike the GRBM, the mcRBM uses its hidden layer to independently parametrize both the mean and covariance of the data through two sets of hidden units. The mcRBM is a combination of the covariance RBM (cRBM) BIB005 , which models the conditional covariance, and the GRBM, which captures the conditional mean. While the GRBM has shown considerable potential as the basis of a highly successful phoneme recognition system BIB006 , it seems that due to difficulties in training the mcRBM, the model has been largely superseded by the mPoT model. The mPoT model (mean-product of Student's T-distributions model) BIB007 is a combination of the GRBM and the product of Student's T-distributions model BIB001 . It is an energy-based model where the conditional distribution over the visible units conditioned on the hidden variables is a multivariate Gaussian (nondiagonal covariance) and the complementary conditional distribution over the hidden variables given the visibles are a set of independent Gamma distributions. The PoT model has recently been generalized to the mPoT model BIB007 to include nonzero Gaussian means by the addition of GRBM-like hidden units, similarly to how the mcRBM generalizes the cRBM. The mPoT model has been used to synthesize large-scale natural images BIB007 that show large-scale features and shadowing structure. It has been used to model natural textures BIB009 in a tiledconvolution configuration (see Section 11.2). Another recently introduced RBM-based model with the objective of having the hidden units encode both the mean and covariance information is the spike-and-slab restricted Boltzmann machine (ssRBM) BIB008 ,  . The ssRBM is defined as having both a real-valued "slab" variable and a binary "spike" variable associated with each unit in the hidden layer. The ssRBM has been demonstrated as a feature learning and extraction scheme in the context of CIFAR-10 object classification  from natural images and has performed well in the role BIB008 ,  . When trained convolutionally (see Section 11.2) on full CIFAR-10 natural images, the model demonstrated the ability to generate natural image samples that seem to capture the broad statistical structure of natural images better than previous parametric generative models, as illustrated with the samples of Fig. 2 . The mcRBM, mPoT, and ssRBM each set out to model real-valued data such that the hidden units encode not only the conditional mean of the data but also its conditional covariance. Other than differences in the training schemes, the most significant difference between these models is how they encode their conditional covariance. While the mcRBM and the mPoT use the activation of the hidden units to enforce constraints on the covariance of x, the ssRBM uses the hidden unit to pinch the precision matrix along the direction specified by the corresponding weight vector. These two ways of modeling conditional covariance diverge when the dimensionality of the hidden layer is significantly different from that of the input. In the overcomplete setting, sparse activation with the ssRBM parametrization permits variance only in the select directions of the sparsely activated hidden units. This is a property the ssRBM shares with sparse coding models BIB002 , BIB003 . On the other hand, in the case of the mPoT or mcRBM, an overcomplete set of constraints on the covariance implies that capturing arbitrary covariance along a particular direction of the input requires decreasing potentially all constraints with positive projection in that direction. This perspective would suggest that the mPoT and mcRBM do not appear to be well suited to provide a sparse representation in the overcomplete setting.