Comparing the inferences of diverse candidate models is an essential part of model checking and escaping local optima. To enable efficient comparison, we introduce an amortized variational inference framework that can perform fast and reliable posterior estimation across models of the same architecture. Our Any Parameter Encoder (APE) extends the encoder neural network common in amortized inference to take both a data feature vector and a model parameter vector as input. APE thus reduces posterior inference across unseen data and models to a single forward pass. In experiments comparing candidate topic models for synthetic data and product reviews, our Any Parameter Encoder yields comparable posteriors to more expensive methods in far less time, especially when the encoder architecture is designed in model-aware fashion. We consider the problem of approximate Bayesian inference for latent variable models, such as topic models (Blei et al., 2003) , embedding models (Mohamed et al., 2009) , and dynamical systems models (Shumway and Stoffer, 1991 ). An important step in using such probabilistic models to extract insight from large datasets is model checking and comparison. While many types of comparison are possible (Gelman et al., 2013) , we focus on a problem that we call within-model comparison. Given several candidate parameter vectors θ 1 , θ 2 , . . ., all from the same space Θ ⊆ R D , our goal is to efficiently determine which parameter θ m is best at explaining a given dataset of N examples {x n } N n=1 . Multiple ways exist to rank candidate parameters, including performance on heldout data or human-in-the-loop inspection. A principled choice is to select the parameter that maximizes the data's marginal likelihood: N n=1 log p(x n |θ m ). For our latent variable models of interest, computing this likelihood requires marginalizing over a hidden variable h n : p(x n |θ m ) = p(x n |h n , θ m )p(h n |θ m )dh n . This integral is challenging even for a single example n and model m. One promising solution is variational inference (VI). Using VI, we can estimate an approximate posterior q(h n |x n , θ m ) over hidden variables. Approximate posteriors q can be used to compute lower bounds on marginal likelihood, and can also be helpful for human inspection of model insights and uncertainties. However, it is expensive to estimate a separate q at each example n and model m. In this paper, we develop new VI tools 1 that enable rapid-yet-effective within-model comparisons for large datasets. The need for within-model comparison (and our methods) is present in many practical modeling tasks. Here we discuss two possible scenarios, with some details specialized to our intended topic modeling applications (Blei, 2012) . First, in human-in-the-loop scenarios, a domain expert may inspect some estimated parameter θ and then suggest an alternative parameter θ that improves interpretability. In topic modeling, this may mean removing "intruder words" to make topics more coherent (Chang et al., 2009) . Second, in automated parameter learning scenarios, many algorithms propose data-driven transformations of the current solution θ into a new candidate θ , in order to escape the local optima common in non-convex optimization objectives for latent variable models (Roberts et al., 2016) , Examples include split-merge proposal moves (Ueda and Ghahramani, 2002; Jain and Neal, 2004) or evolutionary algorithms (Sundararajan and Mengshoel, 2016) . Across both these scenarios, new candidates θ arise repeatedly over time, and estimating approximate posteriors for each is essential to assess fitness yet expensive to perform for large datasets. Our contribution is the Any Parameter Encoder (APE), which amortizes posterior inference across models θ m and data x n . We are inspired by efforts to scale a single model to large datasets by using an encoder neural network (NN) to amortize posterior inference across data examples (Rezende et al., 2014; Kingma and Welling, 2014) . Our key idea is that to additionally generalize across models, we feed model parameter vector θ m and data feature vector x n as input to the encoder. APE is applicable to any model with continuous hidden variables for which amortized inference is possible via the reparameterization trick. Across two datasets and many model parameters, our Any Parameter Encoder produces posterior approximations that are nearly as good as expensive VI, but over 100x faster. Future opportunities include simultaneous training of parameters and encoders, and handling Bayesian nonparametric models where θ changes size during training (Hughes et al., 2015 n )) (6) For encoder methods, the parameters {µ n , log σ 2 n } are the output of a shared encoder NN. For VI, these are free parameters of the optimization problem. Variational Inference (VI). We perform using gradient ascent to maximize the objective in Eq. (1), learning a per-example mean and variance variational parameter. We run gradient updates until our moving average loss (window of 10 steps) has improved by less than 0.001% of its previous value. For our VI runs from random initializations, we use the Adam optimizer with an initial learning rate of .01, decaying the rate by 50% every 5000 steps. For our warm-started runs, we use an initial learning rate of 0.0005. In practice, we ran VI multiple times with different learning rate parameters and took the best one. Table  1 only reports the time to run the best setting, not the total time which includes various restarts. Standard encoder. We use a standard encoder that closely matches the VAE for topic models in Srivastava and Sutton (2017) . The only architectural difference is the addition of a temperature parameter on the µ n vector before applying the softmax to ensure the means lie on the simplex. We found that the additional parameter sped up training by allowing the peakiness of the posterior to be directly tuned by a single parameter. We use a feedforward encoder with two hidden layers, each 100 units. We chose the architecture via hyperparameter sweeps. The total number of trainable parameters in the model is 24,721 on the synthetic data and 316,781 on the real data; this is compared to 16,721 and 19,781 parameters for model-aware APE. NUTS. For the Hamiltonian Monte Carlo (HMC) with the No U-Turn Sampler (NUTS) (Hoffman and Gelman, 2014) , we use a step size of 1 adapted during the warmup phase using Dual Averaging scheme. Upon inspection, we find that the method's slightly lower posterior predictive log likelihood relative to VI is due to its wider posteriors. We also find that the Pyro implementation is (understandably) quite slow and consequently warm-start the NUTS sampler using VI to encourage rapid mixing. We are aware that there exist faster, more specialized implementations, but we decided to keep our tooling consistent for scientific purposes.
We develop VAEs where the encoder takes a model parameter vector as input, so we can do rapid inference for many models
{"src": "scitldr-AIC_train", "paper_id": "rylGty24YB"}