A Survey of Optimization Methods from a Machine Learning Perspective <s> D. Optimization in Variational Inference <s> This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields). We present a number of examples of graphical models, including the QMR-DT database, the sigmoid belief network, the Boltzmann machine, and several variants of hidden Markov models, in which it is infeasible to run exact inference algorithms. We then introduce variational methods, which exploit laws of large numbers to transform the original graphical model into a simplified graphical model in which inference is efficient. Inference in the simpified model provides bounds on probabilities of interest in the original model. We describe a general framework for generating variational transformations based on convex duality. Finally we return to the examples and demonstrate how variational algorithms can be formulated in each case. <s> BIB001 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> D. Optimization in Variational Inference <s> We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet allocation and the hierarchical Dirichlet process topic model. Using stochastic variational inference, we analyze several large collections of documents: 300K articles from Nature, 1.8M articles from The New York Times, and 3.8M articles from Wikipedia. Stochastic inference can easily handle data sets of this size and outperforms traditional variational inference, which can only handle a smaller subset. (We also show that the Bayesian nonparametric topic model outperforms its parametric counterpart.) Stochastic variational inference lets us apply complex Bayesian models to massive data sets. <s> BIB002 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> D. Optimization in Variational Inference <s> ABSTRACTOne of the core problems of modern statistics is to approximate difficult-to-compute probability densities. This problem is especially important in Bayesian statistics, which frames all inference about unknown quantities as a calculation involving the posterior density. In this article, we review variational inference (VI), a method from machine learning that approximates probability densities through optimization. VI has been used in many applications and tends to be faster than classical methods, such as Markov chain Monte Carlo sampling. The idea behind VI is to first posit a family of densities and then to find a member of that family which is close to the target density. Closeness is measured by Kullback–Leibler divergence. We review the ideas behind mean-field variational inference, discuss the special case of VI applied to exponential family models, present a full example with a Bayesian mixture of Gaussians, and derive a variant that uses stochastic optimization to scale up to massive data... <s> BIB003
In the machine learning community, there are many attractive probabilistic models but with complex structure and intractable posteriors, and thus some approximate methods are used, such as variational inference and Markov chain Monte Carlo (MCMC) sampling. Variational inference, a common technique in machine learning, is widely used to approximate the posterior density of the Bayesian model, which transforms intricate inference problems into high-dimensional optimization problems BIB001 ,  . Compared with MCMC, the variational inference is faster and more suitable for dealing with large-scale data. Variational inference has been applied to large-scale machine learning tasks, such as large-scale document analysis, computer vision and computational neuroscience BIB003 . Variational inference often defines a flexible family of distributions indexed by free parameters on latent variables BIB001 , and then finds the variational parameters by solving an optimization problem. Now let us review the principle of variational inference BIB002 . Variational inference approximates the true posterior by attempting to minimize the Kullback-Leibler (KL) divergence between a potential factorized distribution and the true posterior. Let Z = {z i } represent the set of all latent variables and parameters in the model and X = {x i } be a set of all observed data. The joint likelihood of X and Z is p(Z, X) = p(Z)p(X|Z). In Bayesian models, the posterior distributions p(Z|X) should be computed to make further inference. What we need to do is to approximate p(Z|X) with the distribution q(Z) that belongs to a constrained family of distributions. The goal is to make the two distributions as similar as possible. Variational inference chooses KL divergence to measure the difference between the two distributions, that is to minimize the KL divergence of q(Z) and p(Z|X). Here is the formula for the KL divergence between q and p: where log p(X) is replaced by a constant because we are only interested in q. With the above formula, we can know KL divergence is difficult to optimize because it requires knowing the distribution that we are trying to approximate. An alternative method is to maximize the evidence lower bound (ELBO), a lower bound on the logarithm of the marginal probability of the observations. We can obtain ELBO's formula as Variational inference can be treated as an optimization problem with the goal of minimizing the evidence lower bound and the mean-field method. A direct method is to solve this optimization problem using the coordinate ascent, which is called coordinate ascent variational inference (CAVI). CAVI iteratively optimizes each factor of the mean-field variational density, while holding the others fixed BIB003 . Specifically, variational distribution q has the structure of the mean-field, i.e., q(Z) = M i=1 q i (z i ). With this assumption, we can bring the distribution q into the ELBO, by some derivation according to  , and obtain the following formula: Then the CAVI algorithm can be given below in Algorithm 8.