  Variational Auto-Encoders (VAEs) are designed to capture compressible information about a dataset.   As a consequence the information stored in the latent space is seldom sufficient to reconstruct a particular image.   To help understand the type of information stored in the latent space we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space. This allows us to ''imagine'' the information captured in the latent space.   We argue that this is necessary to make a VAE into a truly generative model.   We use our GAN to visualise the latent space of a standard VAE and of a $\beta$-VAE. Variational auto-encoders (VAEs) have made a significant impact since their introduction by Kingma and Welling (2014) . However, one of their perceived problems is their reconstruction performance. This has spawned a wave of research into trying to improve the reconstruction performance (Zhao et al., 2017; Dai and Wipf, 2019; Larsen et al., 2016; Gao et al., 2017; Brock et al., 2017) . We argue that such attempts are misguided. The whole point of VAEs is to capture only compressible information and discard information specific to any particular image. This is a consequence of the well known evidence lower bound or ELBO objective function consisting of a negative log-probability of generating the original image from the latent representation (this is often implemented as a mean squared error between the image and the reconstruction, although as we argue in Appendix A this term should be proportional to the logarithm of the mean squared error) and a KL-divergence between the probability distribution representing a latent code and a 'prior distribution' (usually taken as a multivariate normal with mean zero and unit variance). These two terms have a nice interpretation in terms of the minimum description length (Rissanen, 1978 )-this has been described elsewhere, for example, Chen et al. (2016) . The KL-term can be viewed as a measure of the amount of information in the latent code while the log-probability of the image measures the amount of information required to change the image produced by the decoder into the input image (see Section 3 for details). That is, the latent space of a VAE can be viewed as a model of the dataset-capturing compressible information while not encoding any image specific information (which is cheaper to communicate using the reconstruction loss). The great strength of a VAE is that it builds a model of the dataset that does not over-fit (i.e. code for in-compressible features found in specific images). However, because of this it typically will not do a good job of reconstructing images as the latent code does not contain enough information to do the reconstruction (for very restrictive dataset such as MNIST and Celeb-A a lot of information can be captured in the latent space, but for more complex datasets like ImageNet or CIFAR the reconstructions are poor). Of course, if you want good reconstructions on the training set then the simplest solution is to remove the KL-divergence term and just use an autoencoder. However, having a model that does not over-fit the dataset can be useful, but in this case the decoder of a standard VAE should not be regarded as a generative model-that is not its purpose. If we wish to generate realistic looking images we need to imagine the information discarded by the encoder. As a rather simplified analogy, consider a verbal description of an image "a five year old girl in a blue dress standing on a beach". If we asked different artists to depict such scene there is clearly not enough information to provide pixel-wise or feature-wise similarity between their interpretation although each artist could render a convincing image that satisfies the description. In a similar manner if we want a VAE to act as a generative model we need to build a renderer that will imagine an image consistent with the latent variable representation. A simple way to achieve this is using a modified Generative Adversarial Network (GAN). We call such a model a latent space renderer-GAN (or LSR-GAN). To generate an image we choose a latent vector z from the prior distribution for the VAE. This is passed to a generator network that generates an image,x, with the same dimensions as that of the dataset used to train the VAE. The generated image has both to convince a discriminator network that it is a real image-as is usual for a GAN (Goodfellow et al., 2014) -at the same time the VAE encoder should mapx close to z. To accomplish this we add an additional cost to the normal GAN loss function for the generator (L GEN ) where q φ (·|x) is the probability distribution generated by the VAE encoder given an imagex and z is the latent vector that was put into the GAN generator. Note that when training the LSR-GAN we freeze the weights of the VAE encoder. The constant λ is an adjustable hyperparameter providing a trade-off between how realistic the image should look and how closely it captures the information in the latent space. This modification of the objective function can clearly be applied to any GAN or used with any VAE. Although the idea is simple, it provides a powerful method for visualising (imagining) the information stored in a latent space. Interestingly, it also appears to provide a powerful regularisation mechanism to stabilize the training for GANs. Combinations of VAEs and GANs are, of course, not new (Makhzani et al., 2016; Larsen et al., 2016; Brock et al., 2017; Huang et al., 2018; Srivastava et al., 2017) . In all cases we are aware of GANs have been combined with VAEs to "correct" for the poor reconstruction performance of the VAE (see Appendix B for a more detailed discussion of the literature on VAE-GAN hybrids). As we have argued (and expound on in more detail in Section 3), we believe that the decoder of a VAE does the job it is designed to do. They cannot reconstruct images accurately, because the latent space of a VAE loses information about the image, by design. All we can do is imagine the type of image that a point in the latent space represents. In the next section, we show examples of images generated by the LSR-GAN for both normal VAEs and β-VAEs (we also spend time describing VAEs, β-VAEs and the LSR-GAN in more detail). In addition, in this section we present a number of systematic experiments showing the performance of a VAE and LSR-GAN. In Section 3, we revisit the minimum description length formalism to explain why we believe a VAE is doomed to fail as a generative model. We conclude in Section 4. We cover more technical aspects in the appendices. In Appendix A we show that the correct loss function for a VAE requires minimising a term proportional to the logarithm of the mean squared error. In Appendix B we draw out the similarities and differences between our approach to hybridising VAEs with GANs and other work in this area. We present some additional experimental results in Appendix C. A detailed description of the architecture of LSR-GAN is given in Appendix D. We end the paper with Appendix E by showing some samples generated by randomly drawing latent variables and feeding them to the LSR-GAN. VAEs are often taken to be a pauper's GAN. That is, a method for generating samples that is easier to train than a GAN, but gives slightly worse results. If this is the only objective then it is clearly legitimate to modify the VAE in anyway that will improve its performance. However, we believe that this risks losing one of their most desirable properties, namely their ability to learn features of the whole dataset while avoiding encoding information specific to particular images. We have argued that because of this property, a VAE is not an ideal generative model. It will not be able to reconstruct data accurately and consequently will struggle even more with generating new samples. One of the weaknesses of the vast literature on VAEs is that it often attempts to improve them without regard to what makes VAEs special. As we have argued in this paper, a consistent way of using the latent space of a VAE is to use a GAN as a data renderer, using the VAE encoder to ensure that the GAN is generating images that represent the information encoded in the VAE's latent space. This involves "imagining" the information that the VAE disregards. LSR-GAN can be particularly useful in generating random samples, although, as shown in Appendix E, for very diverse datasets the samples are often not recognisable as real world objects. Although there are already many VAE-GAN hybrids, to the best of our knowledge, they are all designed to "fix" the VAE. In our view VAEs are not broken and "fixing" them is actually likely to break them (i.e. by encoding image specific information in the latent space). Although, the main idea in this paper is relatively simple, we believe its main contribution is as a corrective to the swath of literature on VAEs that, in our view, often throws the baby out with the bath water in an attempt to fix VAEs despite the fact that perform in exactly the way they were designed to.
To understand the information stored in the latent space, we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space.
{"src": "scitldr-AIC_train", "paper_id": "BJe4PyrFvB"}