A Survey of Optimization Methods from a Machine Learning Perspective <s> Algorithm 8 Coordinate Ascent Variational Inference [182] <s> Let M(x) denote the expected value at level x of the response to a certain experiment. M(x) is assumed to be a monotone function of x but is unknown to the experimenter, and it is desired to find the solution x = θ of the equation M(x) = α, where a is a given constant. We give a method for making successive experiments at levels x1, x2, ··· in such a way that xn will tend to θ in probability. <s> BIB001 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> Algorithm 8 Coordinate Ascent Variational Inference [182] <s> We consider situations where training data is abundant and computing resources are comparatively scarce. We argue that suitably designed online learning algorithms asymptotically outperform any batch learning algorithm. Both theoretical and experimental evidences are presented. <s> BIB002 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> Algorithm 8 Coordinate Ascent Variational Inference [182] <s> We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet allocation and the hierarchical Dirichlet process topic model. Using stochastic variational inference, we analyze several large collections of documents: 300K articles from Nature, 1.8M articles from The New York Times, and 3.8M articles from Wikipedia. Stochastic inference can easily handle data sets of this size and outperforms traditional variational inference, which can only handle a smaller subset. (We also show that the Bayesian nonparametric topic model outperforms its parametric counterpart.) Stochastic variational inference lets us apply complex Bayesian models to massive data sets. <s> BIB003 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> Algorithm 8 Coordinate Ascent Variational Inference [182] <s> We introduce stochastic variational inference for Gaussian process models. This enables the application of Gaussian process (GP) models to data sets containing millions of data points. We show how GPs can be variationally decomposed to depend on a set of globally relevant inducing variables which factorize the model in the necessary manner to perform variational inference. Our approach is readily extended to models with non-Gaussian likelihoods and latent variable models based around Gaussian processes. We demonstrate the approach on a simple toy problem and two real world data sets. <s> BIB004 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> Algorithm 8 Coordinate Ascent Variational Inference [182] <s> Gaussian process classification is a popular method with a number of appealing properties. We show how to scale the model within a variational inducing point framework, outperforming the state of the art on benchmark datasets. Importantly, the variational formulation can be exploited to allow classification in problems with millions of data points, as we demonstrate in experiments. <s> BIB005 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> Algorithm 8 Coordinate Ascent Variational Inference [182] <s> ABSTRACTOne of the core problems of modern statistics is to approximate difficult-to-compute probability densities. This problem is especially important in Bayesian statistics, which frames all inference about unknown quantities as a calculation involving the posterior density. In this article, we review variational inference (VI), a method from machine learning that approximates probability densities through optimization. VI has been used in many applications and tends to be faster than classical methods, such as Markov chain Monte Carlo sampling. The idea behind VI is to first posit a family of densities and then to find a member of that family which is close to the target density. Closeness is measured by Kullback–Leibler divergence. We review the ideas behind mean-field variational inference, discuss the special case of VI applied to exponential family models, present a full example with a Bayesian mixture of Gaussians, and derive a variant that uses stochastic optimization to scale up to massive data... <s> BIB006
Input: p(X, Z), X Output: In traditional coordinate ascension algorithms, the efficiency of processing large data is very low, because each iteration needs to compute all the data, which is very timeconsuming. Modern machine learning models often need to analyze and process large-scale data, which is difficult and costly. Stochastic optimization enables machine learning to be extended on massive data BIB002 . This reminds us of an attractive technique to handle large data sets: stochastic optimization BIB001 , BIB006 ,  . By introducing stochastic optimization into variational inference, the stochastic variational inference (SVI) was proposed BIB003 , in which the exponential family is taken as a typical example. Gaussian process (GP) is an important machine learning method based on statistical learning and Bayesian theory. It is suitable for complex regression problems such as high dimensions, small samples, and nonlinearities. Gaussian process has the advantages of strong generalization ability, flexible nonparametric inference, and strong interpretability. However, the complexity and storage requirements of accurate solution for GP are high, which hinders the development of GP under large-scale data. The stochastic variational inference method introduced in this section can popularize variational inference on large-scale datasets, but it can only be applied to probabilistic models with factorized structures. For Gaussian processes whose observations are correlated with each other, the stochastic variational inference can be adapted by introducing the global inducing variables as variational variables BIB004 , BIB005 . Specifically, the observations are assumed to be conditionally independent given the inducing variables and the variational distribution for the inducing variables is assumed to have an explicit form. Thus, the resulting Gaussian process model can be factorized in a necessary manner, enabling the stochastic variational inference. This method can also be easily extended to models with non-Gaussian likelihood and latent variable models based on Gaussian processes.