Survey on Probabilistic Models of Low-Rank Matrix Factorizations <s> Probabilistic Principal Component Analysis <s> Principal component analysis (PCA) is a ubiquitous technique for data analysis and processing, but one which is not based upon a probability model. In this paper we demonstrate how the principal axes of a set of observed data vectors may be determined through maximum-likelihood estimation of parameters in a latent variable model closely related to factor analysis. We consider the properties of the associated likelihood function, giving an EM algorithm for estimating the principal subspace iteratively, and discuss the advantages conveyed by the definition of a probability density function for PCA. <s> BIB001 </s> Survey on Probabilistic Models of Low-Rank Matrix Factorizations <s> Probabilistic Principal Component Analysis <s> Principal component analysis (PCA) is one of the most popular techniques for processing, compressing, and visualizing data, although its effectiveness is limited by its global linearity. While nonlinear variants of PCA have been proposed, an alternative paradigm is to capture data complexity by a combination of local linear PCA projections. However, conventional PCA does not correspond to a probability density, and so there is no unique way to combine PCA models. Therefore, previous attempts to formulate mixture models for PCA have been ad hoc to some extent. In this article, PCA is formulated within a maximum likelihood framework, based on a specific form of gaussian latent variable model. This leads to a well-defined mixture model for probabilistic principal component analyzers, whose parameters can be determined using an expectationmaximization algorithm. We discuss the advantages of this model in the context of clustering, density modeling, and local dimensionality reduction, and we demonstrate its application to image compression and handwritten digit recognition. <s> BIB002 </s> Survey on Probabilistic Models of Low-Rank Matrix Factorizations <s> Probabilistic Principal Component Analysis <s> Abstract We review two similar concepts of hierarchical rank of tensors (which extend the matrix rank to higher order tensors): the TT-rank and the H-rank (hierarchical or H-Tucker rank). Based on this notion of rank, one can define a data-sparse representation of tensors involving O(dnk + dk^3) data for order d tensors with mode sizes n and rank k. Simple examples underline the differences and similarities between the different formats and ranks. Finally, we derive rank bounds for tensors in one of the formats based on the ranks in the other format. <s> BIB003 </s> Survey on Probabilistic Models of Low-Rank Matrix Factorizations <s> Probabilistic Principal Component Analysis <s> Learning about users’ utilities from preference, discrete choice or implicit feedback data is of integral importance in e-commerce, targeted advertising and web search. Due to the sparsity and diffuse nature of data, Bayesian approaches hold much promise, yet most prior work does not scale up to realistic data sizes. We shed light on why inference for such settings is computationally difficult for standard machine learning methods, most of which focus on predicting explicit ratings only. To simplify the difficulty, we present a novel expectation maximization algorithm, driven by expectation propagation approximate inference, which scales to very large datasets without requiring strong factorization assumptions. Our utility model uses both latent bilinear collaborative filtering and non-parametric Gaussian process (GP) regression. In experiments on large real-world datasets, our method gives substantially better results than either matrix factorization or GPs in isolation, and converges significantly faster. <s> BIB004
In Equation BIB003 , the low-dimensional representation z i is an unknown and deterministic parameter vector. In contrast, the original probabilistic PCA BIB001 regards z i as a stochastic vector. This probability model provides a general form of decomposing a d-dimensional input sample x: where the latent random vector z ∼ N (0, I r ), the noise ε ∼ N (0, σ 2 I d ) and µ is the mean vector. In the following, a Maximum Likelihood (ML) method is proposed to obtain the point estimations of W, µ and σ 2 . It is obvious that p(x|z) = N (x|Wz + µ, σ 2 I d ). Hence we have Given the observed sample set X = {x 1 , x 2 , . . . , x N }, the log-likelihood function is The optimal W, µ, σ 2 can be obtained by maximizing L(W, µ, σ 2 |X). By letting the partial derivative of L(W, µ, σ 2 |X) with respect to µ be the zero vector, we can easily get the maximum likelihood estimation of µ: µ ML = x. Hence, the optimal W and σ 2 can be achieved by the stationary points of L(W, µ ML , σ 2 |X) with respect to W and σ 2 . The aforementioned method is slightly complex when computing W and σ 2 . For this purpose, an Expectation Maximization (EM) algorithm was also presented to solve probabilistic PCA. If W, µ and σ 2 are given, then the joint probability distribution of z and x can be derived as follows: The posterior distribution of z for given x is Because we have where is the mean of z. Based on the above analysis, we give the complete-data log-likelihood function: where "const" is a term independent of W, µ and σ 2 . In the expectation step, we take expectation on L C (W, µ, σ 2 |X) with respect to z: According to Equation (39), we have E[ In the maximization step, we first obtain the maximum likelihood estimation of µ: µ ML = x by setting the partial derivative of E z L C (W, µ, σ 2 |X) with respect to µ be a zero vector. Similarly, we can also obtain the optimal estimations of W and σ 2 by solving the equation set: Moreover, a mixture model for probabilistic PCA was proposed in BIB002 . Khan et al. replaced the Gaussian noise with Gaussian process and incorporated the information of preference pairs into the collaborative filtering BIB004 .