Survey on Probabilistic Models of Low-Rank Matrix Factorizations <s> Variational Bayesian Inference <s> We define a new distance measure the resistor-average distance between two probability distributions that is closely related to the Kullback-Leibler distance. While the KullbackLeibler distance is asymmetric in the two distributions, the resistor-average distance is not. It arises from geometric considerations similar to those used to derive the Chernoff distance. Determining its relation to well-known distance measures reveals a new way to depict how commonly used distance measures relate to each other. <s> BIB001 </s> Survey on Probabilistic Models of Low-Rank Matrix Factorizations <s> Variational Bayesian Inference <s> Mean-field variational inference is a method for approximate Bayesian posterior inference. It approximates a full posterior distribution with a factorized set of distributions by maximizing a lower bound on the marginal likelihood. This requires the ability to integrate a sum of terms in the log joint likelihood using this factorized distribution. Often not all integrals are in closed form, which is typically handled by using a lower bound. We present an alternative algorithm based on stochastic optimization that allows for direct optimization of the variational lower bound. This method uses control variates to reduce the variance of the stochastic search gradient, in which existing lower bounds can play an important role. We demonstrate the approach on two non-conjugate models: logistic regression and an approximation to the HDP. <s> BIB002 </s> Survey on Probabilistic Models of Low-Rank Matrix Factorizations <s> Variational Bayesian Inference <s> We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet allocation and the hierarchical Dirichlet process topic model. Using stochastic variational inference, we analyze several large collections of documents: 300K articles from Nature, 1.8M articles from The New York Times, and 3.8M articles from Wikipedia. Stochastic inference can easily handle data sets of this size and outperforms traditional variational inference, which can only handle a smaller subset. (We also show that the Bayesian nonparametric topic model outperforms its parametric counterpart.) Stochastic variational inference lets us apply complex Bayesian models to massive data sets. <s> BIB003 </s> Survey on Probabilistic Models of Low-Rank Matrix Factorizations <s> Variational Bayesian Inference <s> Principal component analysis (PCA) approximates a data matrix with a low-rank one by imposing sparsity on its singular values. Its robust variant can cope with spiky noise by introducing an element-wise sparse term. In this paper, we extend such sparse matrix learning methods, and propose a novel framework called sparse additive matrix factorization (SAMF). SAMF systematically induces various types of sparsity by a Bayesian regularization effect, called model-induced regularization. Although group LASSO also allows us to design arbitrary types of sparsity on a matrix, SAMF, which is based on the Bayesian framework, provides inference without any requirement for manual parameter tuning. We propose an efficient iterative algorithm called the mean update (MU) for the variational Bayesian approximation to SAMF, which gives the global optimal solution for a large subset of parameters in each step. We demonstrate the usefulness of our method on benchmark datasets and a foreground/background video separation problem. <s> BIB004 </s> Survey on Probabilistic Models of Low-Rank Matrix Factorizations <s> Variational Bayesian Inference <s> The variational Bayesian (VB) approximation is known to be a promising approach to Bayesian estimation, when the rigorous calculation of the Bayes posterior is intractable. The VB approximation has been successfully applied to matrix factorization (MF), offering automatic dimensionality selection for principal component analysis. Generally, finding the VB solution is a nonconvex problem, and most methods rely on a local search algorithm derived through a standard procedure for the VB approximation. In this paper, we show that a better option is available for fully-observed VBMF--the global solution can be analytically computed. More specifically, the global solution is a reweighted SVD of the observed matrix, and each weight can be obtained by solving a quartic equation with its coefficients being functions of the observed singular value. We further show that the global optimal solution of empirical VBMF (where hyperparameters are also learned from data) can also be analytically computed. We illustrate the usefulness of our results through experiments in multi-variate analysis. <s> BIB005
Another widely used class of approximating the marginal probability distribution p(z i ) is the variational Bayesian (VB) inference [54] . We still use the notation X = (x 1 , x 2 , . . . , x N ) to represent a matrix composed of N observed datum and z = (z 1 , z 2 , . . . , z M ) to represent a vector constructed by M blocks of latent variables. In VB methods, both X and z are assumed to be stochastic. Given the prior distribution p(z) and the data likelihood p(X|z), the probability distribution of data matrix X, also called the evidence, can be calculated as p(X) = p(z)p(X|z)dz. Then we derive the conditional probability distribution of z via Bayes' rule: However, the integrating term p(z)p(X|z)dz is analytically intractable under normal conditions. Now, we harness VB methods to approximate the posterior distribution p(z|X) as well as p(X). Let q(z) be the trial probability distribution of p(z|X). The log of the probability distribution p(X) is decomposed as below: where dz. The term KL(q||p) is called the Kullback-Leibler (KL) divergence [54,  [85] BIB001  between q(z) and p(z|X), and L(q) is the lower bound of ln p(X) that achieves its lower bound if and only if KL(q||p) = 0 (or equivalently q = p). The above divergence can also be explained as the information gain changing from the prior q(z) to the posterior p(z|X) [85] . Another perspective for Equation (23) is that, The above inequality is obtained by Jensen's inequality. The negative lower bound −L(q) is called the free energy BIB004 BIB005 . The KL divergence KL(q||p) can be regarded as a metric for evaluating the approximation performance of the prior distribution q(z) over the posterior distribution p(z|X) [54] . In the light of Equation (23), minimizing KL(q||p) is equivalent to maximizing L(q). What's more, the lower bound can be further derived as below: The above Equation means that ln p(X) can also be regarded as the expectation of the log likelihood function ln p(X|z) with respect to z. To reduce the difficulty of approximating the trial distribution q(z), we partition all latent variables into M disjoint block variables {z 1 , z 2 , . . . , z M } and assume that z 1 , z 2 , . . . , z M are mutually independent. Based on the above assumption, we have Afterwards, we utilize the mean field theory to obtain the approximation of q(z). Concretely speaking, we update each q(z i ) in turn. Let q(z i ) be unknown and q(z j )(j = i) be fixed. Under this circumstance, we can get the optimal q(z i ) by solving the following variational maximization problem: Plugging Equation (26) into the lower bound of ln p(X), we have where "const" is a term independent of z i and p(X, z i ) is a new probability distribution whose log is defined by ln p(X, Therefore, p(X, z i ) is the optimal solution of problem  obtained by minimizing the KL divergence between q(z i ) and p(X, z i ). An advantage of the VB methods is that they are immune to over fitting. But they also have some shortcomings. For example, the probability distributions derived via VB have always less probability mass in the wings than the true solution and this systematic bias may break applications. A VB method approximates a full posterior distribution by maximizing the corresponding lower bound on the marginal likelihood and it can only handle a smaller dataset. In contrast, the stochastic variational inference optimizes a subset at each iteration. This batch inference algorithm is scalable and outperforms traditional variational inference methods BIB002 BIB003 .