We propose a new form of an autoencoding model which incorporates the best properties of variational autoencoders (VAE) and generative adversarial networks (GAN). It is known that GAN can produce very realistic samples while VAE does not suffer from mode collapsing problem. Our model optimizes λ-Jeffreys divergence between the model distribution and the true data distribution. We show that it takes the best properties of VAE and GAN objectives. It consists of two parts. One of these parts can be optimized by using the standard adversarial training, and the second one is the very objective of the VAE model. However, the straightforward way of substituting the VAE loss does not work well if we use an explicit likelihood such as Gaussian or Laplace which have limited flexibility in high dimensions and are unnatural for modelling images in the space of pixels. To tackle this problem we propose a novel approach to train the VAE model with an implicit likelihood by an adversarially trained discriminator. In an extensive set of experiments on CIFAR-10 and TinyImagent datasets, we show that our model achieves the state-of-the-art generation and reconstruction quality and demonstrate how we can balance between mode-seeking and mode-covering behaviour of our model by adjusting the weight λ in our objective. Variational autoencoder (VAE) (Kingma et al., 2014; Rezende et al., 2014; Titsias & Lázaro-Gredilla, 2014 ) is one of the most popular approaches for modeling complex high-dimensional distributions. It has been applied successfully to many practical problems. It has several nice properties such as learning low-dimensional representations for the objects and ability to conditional generation. Due to an explicit reconstruction term in its objective, one may ensure that VAE can generate all objects from the training set. These advantages, however, come at a price. It is a known fact that VAE tends to generate unrealistic objects, e.g., blurred images. Such behaviour can be explained by the properties of a maximum likelihood estimation (MLE) which is used to fit a restricted VAE model p θ (x) in data that comes from a complex distribution p with an equiprobable mixture of two Gaussians with learnable location and scale. Plots a)-c) show pairwise comparisons of optimal log-densities, the plot d) compares optimal densities themselves. ). This way, we encourage our model to be mode-seeking while still having relatively high values of p θ (x) on all objects from a training set, thus preventing the mode-collapse. We note that J λ (p θ (x) p * (x)) is not symmetric with respect to p θ (x) and p * (x) and by the weight λ we can balance between mode-seeking and mass-covering behaviour. However, the straightforward way of substituting each KL term with GAN and VAE losses does not work well in practice if we use an explicit likelihood for object reconstruction in VAE objective. Such simple distributions as Gaussian or Laplace that are usually used in VAE have limited flexibility and are unnatural for modelling images in the space of pixels. To tackle this problem we propose a novel approach to train the VAE model in an adversarial manner. We show how we can estimate the implicit likelihood in our loss function by an adversarially trained discriminator. We theoretically analyze the introduced loss function and show that under assumptions of optimal discriminators, our model minimizes the λ-Jeffreys divergence J λ (p θ (x) p * (x)) and we call our method as Implicit λ-Jeffreys Autoencoder (λ-IJAE). In an extensive set of experiments, we evaluate the generation and reconstruction ability of our model on CIFAR10 (Krizhevsky et al., 2009) and TinyImagenet datasets. It shows the state-of-the-art trade-off between generation and reconstruction quality. We demonstrate how we can balance between the ability of generating realistic images and the reconstruction ability by changing the weight λ in our objective. Based on our experimental study we derive a default choice for λ that establishes a reasonable compromise between mode-seeking and mass-covering behaviour of our model and this choice is consistent over these two datasets. In the paper, we considered a fusion of VAE and GAN models that takes the best of two worlds: it has sharp and coherent samples and can encode observations into low-dimensional representations. We provide a theoretical analysis of our objective and show that it is equivalent to the Jeffreys divergence. In experiments, we demonstrate that our model achieves a good balance between generation and reconstruction quality. It confirms our assumption that the Jeffreys divergence is the right choice for learning complex high-dimensional distributions in the case of the limited capacity of the model. Proof. Now we will show the second term is equal to zero given our assumptions: Where we have used the (1) and (2) properties of the likelihoods r(x|y) (Definition 1): To generate the plot 1 we considered the following setup: a target distribution was a mixture: While the model as an equiprobable mixture of two learnable Gaussians: The optimal θ was found by making 10,000 stochastic gradient descent iterations on Monte Carlo estimations of the corresponding divergences with a batch size of 1000. We did 50 independent runs for each method to explore different local optima and chose the best one based on a divergence estimate with 100,000 samples Monte Carlo samples.
We propose a new form of an autoencoding model which incorporates the best properties of variational autoencoders (VAE) and generative adversarial networks (GAN)
{"src": "scitldr-AIC_train", "paper_id": "Syxc1yrKvr"}