Here we propose a novel model family with the objective of learning to disentangle the factors of variation in data. Our approach is based on the spike-and-slab restricted Boltzmann machine which we generalize to include higher-order interactions among multiple latent variables. Seen from a generative perspective, the multiplicative interactions emulates the entangling of factors of variation. Inference in the model can be seen as disentangling these generative factors. Unlike previous attempts at disentangling latent factors, the proposed model is trained using no supervised information regarding the latent factors. We apply our model to the task of facial expression classification.
also propose to disentangle factors of variation by learning to extract features associated with pose parameters, where the changes in pose parameters (but not the feature values) are known at training time. The proposed model is also closely related to recent work @cite_0 , where higher-order Boltzmann Machines are used as models of spatial transformations in images. While there are a number of differences between this model and ours, the most significant difference is our use of multiplicative interactions between variables. While they included higher-order interactions within the Boltzmann energy function, they were used exclusively between observed variables, dramatically simplifying the inference and learning procedures. Another major point of departure is that instead of relying on low-rank approximations to the weight tensor, our approach employs highly structured and sparse connections between latent variables (e.g. @math is not interact with or @math for @math ), reminiscent of recent work on structured sparse coding @cite_15 and structured @math -norms @cite_21 . As discussed above, our use of a sparse connection structure allows us to isolate groups of interacting latent variables. Keeping the interactions local in this way, is a key component of our ability to successfully learn using only unsupervised data.
@cite_0 @cite_15 @cite_21
2136163184 2161977692 2138265962
To allow the hidden units of a restricted Boltzmann machine to model the transformation between two successive images, Memisevic and Hinton (2007) introduced three-way multiplicative interactions that use the intensity of a pixel in the first image as a multiplicative gain on a learned, symmetric weight between a pixel in the second image and a hidden unit. This creates cubically many parameters, which form a three-dimensional interaction tensor. We describe a low-rank approximation to this interaction tensor that uses a sum of factors, each of which is a three-way outer product. This approximation allows efficient learning of transformations between larger image patches. Since each factor can be viewed as an image filter, the model as a whole learns optimal filter pairs for efficiently representing transformations. We demonstrate the learning of optimal filter pairs from various synthetic and real image sequences. We also show how learning about image transformations allows the model to perform a simple visual analogy task, and we show how a completely unsupervised network trained on transformations perceives multiple motions of transparent dot patterns in the same way as humans. This work describes a conceptually simple method for structured sparse coding and dictionary design. Supposing a dictionary with K atoms, we introduce a structure as a set of penalties or interactions between every pair of atoms. We describe modifications of standard sparse coding algorithms for inference in this setting, and describe experiments showing that these algorithms are efficient. We show that interesting dictionaries can be learned for interactions that encode tree structures or locally connected structures. Finally, we show that our framework allows us to learn the values of the interactions from the data, rather than having them pre-specified. Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. While naturally cast as a combinatorial optimization problem, variable or feature selection admits a convex relaxation through the regularization by the @math -norm. In this paper, we consider situations where we are not only interested in sparsity, but where some structural prior knowledge is available as well. We show that the @math -norm can then be extended to structured norms built on either disjoint or overlapping groups of variables, leading to a flexible framework that can deal with various structures. We present applications to unsupervised learning, for structured sparse principal component analysis and hierarchical dictionary learning, and to supervised learning in the context of non-linear variable selection.