Learning disentangled representations of  data is one of the central themes in unsupervised learning in general and generative modelling in particular.   In this work,  we tackle a slightly more intricate scenario where the observations are generated from a conditional distribution of some known control variate and some latent noise variate.   To this end, we present a hierarchical model and a training method (CZ-GEM) that leverages some of the recent developments in likelihood-based and likelihood-free generative models.   We show that by formulation, CZ-GEM introduces the right inductive biases that ensure the disentanglement of the control from the noise variables, while also keeping the components of the control variate disentangled. This is achieved without compromising on the quality of the generated samples. Our approach is simple, general, and can be applied both in supervised and unsupervised settings. Consider the following scenario: a hunter-gatherer walking in the African Savannah some 50,000 years ago notices a lioness sprinting out of the bush towards her. In a split second, billions of photons reaching her retinas carrying an enormous amount of information: the shade of the lioness' fur, the angle of its tail, the appearance of every bush in her field of view, the mountains in the background and the clouds in the sky. Yet at this point there is a very small number of attributes which are of importance: the type of the charging animal, its approximate velocity and its location. The rest are just details. The significance of the concept that the world, despite its complexity, can be described by a few explanatory factors of variation, while ignoring the small details, cannot be overestimated. In machine learning there is a large body of work aiming to extract low-dimensional, interpretable representations of complex, often visual, data. Interestingly, many of the works in this area are associated with developing generative models. The intuition is that if a model can generate a good approximation of the data then it must have learned something about its underlying representation. This representation can then be extracted either by directly inverting the generative process (Srivastava et al., 2019b) or by extracting intermediate representations of the model itself (Kingma & Welling, 2014; Higgins et al., 2017) . Clearly, just learning a representation, even if it is low-dimensional, is not enough. The reason is that while there could be many ways to compress the information captured in the data, allowing good enough approximations, there is no reason to a priori assume that such a representation is interpretable and disentangled in the sense that by manipulating certain dimensions of the representation one can control attributes of choice, say the pose of a face, while keeping other attributes unchanged. The large body of work on learning disentangled representations tackles this problem in several settings; fully supervised, weakly supervised and unsupervised, depending on the available data (Tran et al., 2018; Reed et al., 2014; Jha et al., 2018; Mathieu et al., 2016; Higgins et al., 2017; Kim & Mnih, 2018; Nguyen-Phuoc et al., 2019; Narayanaswamy et al., 2017) . Ideally, we would like to come up with an unsupervised generative model that can generate samples which approximate the data to a high level of accuracy while also giving rise to a disentangled and interpretable representation. In the last decade two main approaches have captured most of the attention; Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs). In their original versions, both GANs (Goodfellow et al., 2014) and VAEs (Kingma & Welling, 2014) were trained in an unsupervised manner and (a) Chair rotation generated by CGAN (b) Chair rotation generated by CZ-GEM Figure 1 : Changing the azimuth of chairs in CGAN and CZ-GEM while holding Z constant. Unlike CZ-GEM, C and Z are clearly entangled in CGAN as changing C also changes the type of chair even though Z is held constant. gave rise to entangled representations. Over the years, many methods to improve the quality of the generated data as well as the disentanglement of the representations have been suggested (Brock et al., 2018; Kingma & Dhariwal, 2018; Nguyen-Phuoc et al., 2019; Jeon et al., 2018) . By and large, GANs are better than VAEs in the quality of the generated data while VAEs learn better disentangled representations, in particular in the unsupervised setting. In this paper, we present a framework for disentangling a small number of control variables from the rest of the latent space which accounts for all the additional details, while maintaining a high quality of the generated data. We do that by combining VAE and GAN approaches thus enjoying the best of both worlds. The framework is general and works in both the supervised and unsupervised settings. Let us start with the supervised case. We are provided with paired examples (x, c) where x is the observation and c is a control variate. Crucially, there exists a one-to-many map from c to the space of observations, and there are other unobserved attributes z (or noise) that together completely define x. For instance, if x were an image of a single object, c controls the orientation of the object relative to the camera and z could represent object identity, texture or background. Our goal is to learn a generative model p θ (x|c, z) that fulfills two criteria: If we were learning models of images, we would like the generated images to look realistic and match the true conditional distribution p(x|c). We present a simple yet effective method of learning representations in deep generative models in the setting where the observation is determined by control variate C and noise variate Z. Our method ensures that in the learned representation both C and Z are disentangled as well as the components of C themselves. This is done without compromising the quality of the generated samples. In future work, we would like to explore how this method can be applied to input with multiple objects.
Hierarchical generative model (hybrid of VAE and GAN) that learns a disentangled representation of data without compromising the generative quality.
{"src": "scitldr-AIC_train", "paper_id": "r1e74a4twH"}