In this paper, we describe the "implicit autoencoder" (IAE), a generative autoencoder in which both the generative path and the recognition path are parametrized by implicit distributions. We use two generative adversarial networks to define the reconstruction and the regularization cost functions of the implicit autoencoder, and derive the learning rules based on maximum-likelihood learning. Using implicit distributions allows us to learn more expressive posterior and conditional likelihood distributions for the autoencoder. Learning an expressive conditional likelihood distribution enables the latent code to only capture the abstract and high-level information of the data, while the remaining information is captured by the implicit conditional likelihood distribution. For example, we show that implicit autoencoders can disentangle the global and local information, and perform deterministic or stochastic reconstructions of the images. We further show that implicit autoencoders can disentangle discrete underlying factors of variation from the continuous factors in an unsupervised fashion, and perform clustering and semi-supervised learning. Deep generative models have achieved remarkable success in recent years. One of the most successful models is the generative adversarial network (GAN) BID7 , which employs a two player min-max game. The generative model, G, samples the noise vector z ∼ p(z) and generates the sample G(z). The discriminator, D(x), is trained to identify whether a point x comes from the data distribution or the model distribution; and the generator is trained to maximally confuse the discriminator. The cost function of GAN is DISPLAYFORM0 GANs can be viewed as a general framework for learning implicit distributions BID18 BID12 . Implicit distributions are probability distributions that are obtained by passing a noise vector through a deterministic function that is parametrized by a neural network. In the probabilistic machine learning problems, implicit distributions trained with the GAN framework can learn distributions that are more expressive than the tractable distributions trained with the maximum-likelihood framework.Variational autoencoders (VAE) BID13 BID20 are another successful generative models that use neural networks to parametrize the posterior and the conditional likelihood distributions. Both networks are jointly trained to maximize a variational lower bound on the data log-likelihood. One of the limitations of VAEs is that they learn factorized distributions for both the posterior and the conditional likelihood distributions. In this paper, we propose the "implicit autoencoder" (IAE) that uses implicit distributions for learning more expressive posterior and conditional likelihood distributions. Learning a more expressive posterior will result in a tighter variational bound; and learning a more expressive conditional likelihood distribution will result in a global vs. local decomposition of information between the prior and the conditional likelihood. This enables the latent code to only capture the information that we care about such as the high-level and abstract information, while the remaining low-level information of data is separately captured by the noise vector of the implicit decoder.Implicit distributions have been previously used in learning generative models in works such as adversarial autoencoders (AAE) BID16 , adversarial variational Bayes (AVB) (Mescheder et al., 2017) , ALI (Dumoulin et al., 2016) , BiGAN BID5 and other works such as BID12 BID22 . The global vs. local decomposition of information has also been studied in previous works such as PixelCNN autoencoders (van den Oord et al., 2016) , PixelVAE BID9 , variational lossy autoencoders BID4 , PixelGAN autoencoders BID15 , or other works such as BID2 BID8 BID0 . In the next section, we first propose the IAE and then establish its connections with the related works. In this paper, we proposed the implicit autoencoder, which is a generative autoencoder that uses implicit distributions to learn expressive variational posterior and conditional likelihood distributions. We showed that in IAEs, the information of the data distribution is decomposed between the prior and the conditional likelihood. When using a low dimensional Gaussian distribution for the global code, we showed that the IAE can disentangle high-level and abstract information from the low-level and local statistics. We also showed that by using a categorical latent code, we can learn discrete factors of variation and perform clustering and semi-supervised learning.
We propose a generative autoencoder that can learn expressive posterior and conditional likelihood distributions using implicit distributions, and train the model using a new formulation of the ELBO.
{"src": "scitldr-AIC_train", "paper_id": "HyMRaoAqKX"}