Amortized inference has led to efficient approximate inference for large datasets. The quality of posterior inference is largely determined by two factors: a) the ability of the variational distribution to model the true posterior and b) the capacity of the recognition network to generalize inference over all datapoints. We analyze approximate inference in variational autoencoders in terms of these factors. We find that suboptimal inference is often due to amortizing inference rather than the limited complexity of the approximating distribution. We show that this is due partly to the generator learning to accommodate the choice of approximation. Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation. There has been significant work on improving inference in variational autoencoders (VAEs) BID13 BID22 through the development of expressive approximate posteriors BID21 BID14 BID20 BID27 . These works have shown that with more expressive approximate posteriors, the model learns a better distribution over the data.In this paper, we analyze inference suboptimality in VAEs: the mismatch between the true and approximate posterior. In other words, we are interested in understanding what factors cause the gap between the marginal log-likelihood and the evidence lower bound (ELBO). We refer to this as the inference gap. Moreover, we break down the inference gap into two components: the approximation gap and the amortization gap. The approximation gap comes from the inability of the approximate distribution family to exactly match the true posterior. The amortization gap refers to the difference caused by amortizing the variational parameters over the entire training set, instead of optimizing for each datapoint independently. We refer the reader to Table 1 for detailed definitions and FIG0 for a simple illustration of the gaps. In FIG0 , L[q] refers to the ELBO using an amortized distribution q, whereas q * is the optimal q within its variational family. Our experiments investigate how the choice of encoder, posterior approximation, decoder, and model optimization affect the approximation and amortization gaps.We train VAE models in a number of settings on the MNIST, Fashion-MNIST BID30 , and CIFAR-10 datasets.Our contributions are: a) we investigate inference suboptimality in terms of the approximation and amortization gaps, providing insight to guide future improvements in VAE inference, b) we quantitatively demonstrate that the learned true posterior accommodates the choice of approximation, and c) we demonstrate that using parameterized functions to improve the expressiveness of the approximation plays a large role in reducing error caused by amortization. Table 1 : Summary of Gap Terms. The middle column refers to the general case where our variational objective is a lower bound on the marginal log-likelihood (not necessarily the ELBO). The right most column demonstrates the specific case in VAEs. q * (z|x) refers to the optimal approximation within a family Q, i.e. q * (z|x) = arg min q∈Q KL (q(z|x)||p(z|x)). In this paper, we investigated how encoder capacity, approximation choice, decoder capacity, and model optimization influence inference suboptimality in terms of the approximation and amortization gaps. We found that the amortization gap is often the leading source of inference suboptimality and that the generator reduces the approximation gap by learning a true posterior that fits to the choice of approximate distribution. We showed that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation. We confirmed that increasing the capacity of the encoder reduces the amortization error. We also showed that optimization techniques, such as entropy annealing, help the generative model to better utilize the flexibility of the expressive variational distribution. Computing these gaps can be useful for guiding improvements to inference in VAEs. Future work includes evaluating other types of expressive approximations and more complex likelihood functions. The VAE model of FIG1 uses a decoder p(x|z) with architecture: 2 − 100 − 784, and an encoder q(z|x) with architecture: 784 − 100 − 4. We use tanh activations and a batch size of 50. The model is trained for 3000 epochs with a learning rate of 10 −4 using the ADAM optimizer BID12 .
We decompose the gap between the marginal log-likelihood and the evidence lower bound and study the effect of the approximate posterior on the true posterior distribution in VAEs.
{"src": "scitldr-AIC_train", "paper_id": "Bki4EfWCb"}