The standard variational lower bounds used to train latent variable models produce biased estimates of most quantities of interest. We introduce an unbiased estimator of the log marginal likelihood and its gradients for latent variable models based on randomized truncation of infinite series. If parameterized by an encoder-decoder architecture, the parameters of the encoder can be optimized to minimize its variance of this estimator. We show that models trained using our estimator give better test-set likelihoods than a standard importance-sampling based approach for the same average computational cost. This estimator also allows use of latent variable models for tasks where unbiased estimators, rather than marginal likelihood lower bounds, are preferred, such as minimizing reverse KL divergences and estimating score functions. Latent variable models are powerful tools for constructing highly expressive data distributions and for understanding how high-dimensional observations might possess a simpler representation. Latent variable models are often framed as probabilistic graphical models, allowing these relationships to be expressed in terms of conditional independence. Mixture models, probabilistic principal component analysis (Tipping & Bishop, 1999) , hidden Markov models, and latent Dirichlet allocation (Blei et al., 2003) are all examples of powerful latent variable models. More recently there has been a surge of interest in probabilistic latent variable models that incorporate flexible nonlinear likelihoods via deep neural networks (Kingma & Welling, 2014) . These models can blend the advantages of highly structured probabilistic priors with the empirical successes of deep learning (Johnson et al., 2016; Luo et al., 2018) . Moreover, these explicit latent variable models can often yield relatively interpretable representations, in which simple interpolation in the latent space can lead to semantically-meaningful changes in high-dimensional observations (e.g., Higgins et al. (2017) ). It can be challenging, however, to fit the parameters of a flexible latent variable model, since computing the marginal probability of the data requires integrating out the latent variables in order to maximize the likelihood with respect to the model parameters. Typical approaches to this problem include the celebrated expectation maximization algorithm (Dempster et al., 1977) , Markov chain Monte Carlo, and the Laplace approximation. Variational inference generalizes expectation maximization by forming a lower bound on the aforementioned (log) marginal likelihood, using a tractable approximation to the unmanageable posterior over latent variables. The maximization of this lower bound-rather than the true log marginal likelihood-is often relatively straightforward when using automatic differentiation and Monte Carlo sampling. However, a lower bound may be ill-suited for tasks such as posterior inference and other situations where there exists an entropy maximization objective; for example in entropy-regularized reinforcement learning (Williams & Peng, 1991; Mnih et al., 2016; Norouzi et al., 2016) which requires minimizing the log probability of the samples under the model. While there is a long history in Bayesian statistics of estimating the marginal likelihood (e.g., Newton & Raftery (1994) ; Neal (2001)), we often want high-quality estimates of the logarithm of the marginal likelihood, which is better behaved when the data is high dimensional; it is not as susceptible to underflow and it has gradients that are numerically sensible. However, the log transformation introduces some challenges: Monte Carlo estimation techniques such as importance sampling do not straightforwardly give unbiased estimates of this quantity. Nevertheless, there has been significant work to construct estimators of the log marginal likelihood in which it is possible to explicitly trade off between bias against computational cost (Burda et al., 2016; Bamler et al., 2017; Nowozin, 2018) . Unfortunately, while there are asymptotic regimes where the bias of these estimators approaches zero, it is always possible to optimize the parameters to increase this bias to infinity. In this work, we construct an unbiased estimator of the log marginal likelihood. Although there is no theoretical guarantee that this estimator has finite variance, we find that it can work well in practice. We show that this unbiased estimator can train latent variable models to achieve higher test log-likelihood than lower bound estimators at the same expected compute cost. More importantly, this unbiased estimator allows us to apply latent variable models in situations where these models were previously problematic to optimize with lower bound estimators. Such applications include latent variable modeling for posterior inference and for reinforcement learning in high-dimensional action spaces, where an ideal model is one that is highly expressive yet efficient to sample from. We introduced SUMO, a new unbiased estimator of the log probability for latent variable models, and demonstrated tasks for which this estimator performs better than standard lower bounds. Specifically, we investigated applications involving entropy maximization where a lower bound performs poorly, but our unbiased estimator can train properly with relatively smaller amount of compute. In the future, we plan to investigate new families of gradient-based optimizers which can handle heavy-tailed stochastic gradients. It may also be fruitful to investigate the use of convex combination of consistent estimators within the SUMO approach, as any convex combination is unbiased, or to apply variance reduction methods to increase stability of training with SUMO. Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, figshare, 2010. A APPENDIX where z 1 , .. , z k are sampled independently from q(z; x). And we define the k-th term of the infinite . Using the properties of IWAE in equation 6, we have∆ k (x) ≥ 0, and which means the series converges absolutely. This is a sufficient condition for finite expectation of the Russian roulette estimator (Chen et al. (2019) ; Lemma 3). Applying equation 7 to the series: Let , Hence our estimator is constructed: And it can be easily seen from equation 22 and equation 23 that SUMO is an unbiased estimator of the log marginal likelihood: A.2 CONVERGENCE OF ∆ k We follow the analysis of JVI (Nowozin, 2018) , which applied the delta method for moments to show the asymptotic results on the bias and variance of IWAE k both at a rate of O( and we define Y k := 1 k k i=1 w i as the sample mean and we have E[ We note that we rely on ||Y k − µ|| < 1 for this power series to converge. This condition was implicitly assumed, but not explicitly noted, in (Nowozin, 2018) . This condition will hold for sufficiently large k so long as the moments of w i exist: one could bound the probability ||Y k −µ|| ≥ 1 by Chebyshev's inequality or by the Central Limit Theorem. We use the central moments Expanding Eq. 28 to order two gives Since we use cumulative sum to compute Y k and Y k+1 , we obtain We note that Without loss of generality, suppose j ≥ k + 1, For clarity, let C k = Y k − µ be the zero-mean random variable. Nowozin (2018) gives the relations Expanding both the sums inside the brackets to order two: We will proceed by bounding each of the terms (1), (2), (3), (4). First, we decompose C j . Let We know that B k,j is independent of C k and Now we show that (1) is zero: We now investigate (2): We now show that (3) is zero: Finally, we investigate (4): Using the relation in equation 36, we have
We create an unbiased estimator for the log probability of latent variable models, extending such models to a larger scope of applications.
{"src": "scitldr-AIC_train", "paper_id": "SylkYeHtwr"}