We present an information-theoretic framework for understanding trade-offs in unsupervised learning of deep latent-variables models using variational inference. This framework emphasizes the need to consider latent-variable models along two dimensions: the ability to reconstruct inputs (distortion) and the communication cost (rate). We derive the optimal frontier of generative models in the two-dimensional rate-distortion plane, and show how the standard evidence lower bound objective is insufficient to select between points along this frontier. However, by performing targeted optimization to learn generative models with different rates, we are able to learn many models that can achieve similar generative performance but make vastly different trade-offs in terms of the usage of the latent variable. Through experiments on MNIST and Omniglot with a variety of architectures, we show how our framework sheds light on many recent proposed extensions to the variational autoencoder family. Deep learning has led to tremendous advances in supervised learning BID39 BID20 BID46 ; however, unsupervised learning remains a challenging area. Recent advances in variational inference (VI) BID23 BID33 , have led to an explosion of research in the area of deep latent-variable models and breakthroughs in our ability to model natural high-dimensional data. This class of models typically optimize a lower bound on the log-likelihood of the data known as the evidence lower bound (ELBO), and leverage the "reparameterization trick" to make large-scale training feasible.However, a number of papers have observed that VAEs trained with powerful decoders can learn to ignore the latent variables BID44 BID6 . We demonstrate this empirically and explain the issue theoretically by deriving the ELBO in terms of the mutual information between X, the data, and Z, the latent variables. Having done so, we show that the previously-described β-VAE objective (Higgins et al., 2017 ) has a theoretical justification in terms of a Legendre-transformation of a constrained optimization of the mutual information. This leads to the core point of this paper, which is that the optimal rate of information in a model is taskdependent, and optimizing the ELBO directly makes the selection of that rate purely a function of architectural choices, whereas by using β-VAE or other constrained optimization objectives, practitioners can learn models with optimal rates for their particular task without having to do extensive architectural search.Mutual information provides a reparameterization-independent measure of dependence between two random variables. Computing mutual information exactly in high dimensions is problematic BID29 Gao et al., 2017 ), so we turn to recently developed tools in variational inference to approximate it. We find that a natural lower and upper bound on the mutual information between the input and latent variable can be simply related to the ELBO, and understood in terms of two terms: (1) a lower bound that depends on the distortion, or how well an input can be reconstructed through the encoder and decoder, and (2) an upper bound that measures the rate, or how much information is retained about the input. Together these terms provide a unifying perspective on the set of optimal models given a dataset, and show that there exists a continuum of models that make very different trade-offs in terms of rate and distortion.By leveraging additional information about the amount of information contained in the latent variable, we show that we can recover the ground-truth generative model used to create the data in a toy model. We perform extensive experiments on MNIST and Omniglot using a variety of encoder, decoder, and prior architectures and demonstrate how our framework provides a simple and intuitive mechanism for understanding the trade-offs made by these models. We further show that we can control this tradeoff directly by optimizing the β-VAE objective, rather than the ELBO. By varying β, we can learn many models with the same architecture and comparable generative performance (in terms of marginal data log likelihood), but that exhibit qualitatively different behavior in terms of the usage of the latent variable and variability of the decoder. We have motivated the β-VAE objective on information theoretic grounds, and demonstrated that comparing model architectures in terms of the rate-distortion plot offers a much better look at their performance and tradeoffs than simply comparing their marginal log likelihoods. Additionally, we have shown a simple way to fix models that ignore the latent space due to the use of a powerful decoder: simply reduce β and retrain. This fix is much easier to implement than other solutions that have been proposed in the literature, and comes with a clear theoretical justification. We strongly encourage future work to report rate and distortion values independently, rather than just reporting the log likelihood. If future work proposes new architectural regularization techniques, we suggest the authors train their objective at various rate distortion tradeoffs to demonstrate and quantify the region of the RD plane where their method dominates.Through a large set of experiments we have demonstrated the performance at various rates and distortion tradeoffs for a set of representative architectures currently under study, confirming the power of autoregressive decoders, especially at low rates. We have also shown that current approaches seem to have a hard time achieving high rates at low distortion. This suggests a set of experiments with a simple encoder / decoder pair but a powerful autoregressive marginal posterior approximation, which should in principle be able to reach the autoencoding limit, with vanishing distortion and rates approaching the data entropy.Interpreting the β-VAE objective as a constrained optimization problem also hints at the possibility of applying more powerful constrained optimization techniques, which we hope will be able to advance the state of the art in unsupervised representation learning. A RESULTS ON OMNIGLOT FIG4 plots the RD curve for various models fit to the Omniglot dataset BID25 , in the same form as the MNIST results in FIG2 . Here we explored βs for the powerful decoder models ranging from 1.1 to 0.1, and βs of 0.9, 1.0, and 1.1 for the weaker decoder models. On Omniglot, the powerful decoder models dominate over the weaker decoder models. The powerful decoder models with their autoregressive form most naturally sit at very low rates. We were able to obtain finite rates by means of KL annealing. Further experiments will help to fill in the details especially as we explore differing β values for these architectures on the Omniglot dataset. Our best achieved ELBO was at 90.37 nats, set by the ++-model with β = 1.0 and KL annealing. This model obtains R = 0.77, D = 89.60, ELBO = 90.37 and is nearly auto-decoding. We found 14 models with ELBOs below 91.2 nats ranging in rates from 0.0074 nats to 10.92 nats.Similar to FIG3 in FIG5 we show sample reconstruction and generated images from the same "-+v" model family trained with KL annealing but at various βs. Just like in the MNIST case, this demonstrates that we can smoothly interpolate between auto-decoding and auto-encoding behavior in a single model family, simply by adjusting the β value.
We provide an information theoretic and experimental analysis of state-of-the-art variational autoencoders.
{"src": "scitldr-AIC_train", "paper_id": "H1rRWl-Cb"}