Tensors have found application in a variety of fields, ranging from chemometrics to signal processing and beyond. In this paper, we consider the problem of multilinear modeling of sparse count data. Our goal is to develop a descriptive tensor factorization model of such data, along with appropriate algorithms and theory. To do so, we propose that the random variation is best described via a Poisson distribution, which better describes the zeros observed in the data as compared to the typical assumption of a Gaussian distribution. Under a Poisson assumption, we fit a model to observed data using the negative log-likelihood score. We present a new algorithm for Poisson tensor factorization called CANDECOMP--PARAFAC alternating Poisson regression (CP-APR) that is based on a majorization-minimization approach. It can be shown that CP-APR is a generalization of the Lee--Seung multiplicative updates. We show how to prevent the algorithm from converging to non-KKT points and prove convergence of CP-APR under mil...
In terms of convergence, Lin @cite_8 and Gillis and Glienur @cite_28 have shown convergence of two different modified versions of the Lee-Seung method for LS. Finesso and Spreij @cite_21 (tensor extension in @cite_2 ) have shown convergence of the Lee-Seung method for KL divergence; however, we show later that numerical issues arise if the iterates come near to the boundary. This is related to the problems demonstrated by Gonzalez and Zhang @cite_20 that show, in the case of LS loss, the Lee and Seung method can converge to non-KKT points; we show a similar example for KL divergence in misconvergence .
@cite_8 @cite_28 @cite_21 @cite_2 @cite_20
2963956018 2129455341 1982359839 2151454335
We propose a general purpose variational inference algorithm that forms a natural counterpart of gradient descent for optimization. Our method iteratively transports a set of particles to match the target distribution, by applying a form of functional gradient descent that minimizes the KL divergence. Empirical studies are performed on various real world models and datasets, on which our method is competitive with existing state-of-the-art methods. The derivation of our method is based on a new theoretical result that connects the derivative of KL divergence under smooth transforms with Stein’s identity and a recently proposed kernelized Stein discrepancy, which is of independent interest. In this paper we study Nonnegative Tensor Factorization (NTF) based on the Kullback---Leibler (KL) divergence as an alternative Csiszar---Tusnady procedure. We propose new update rules for the aforementioned divergence that are based on multiplicative update rules. The proposed algorithms are built on solid theoretical foundations that guarantee that the limit point of the iterative algorithm corresponds to a stationary solution of the optimization procedure. Moreover, we study the convergence properties of the optimization procedure and we present generalized pythagorean rules. Furthermore, we provide clear probabilistic interpretations of these algorithms. Finally, we discuss the connections between generalized Probabilistic Tensor Latent Variable Models (PTLVM) and NTF, proposing in that way algorithms for PTLVM for arbitrary multivariate probabilistic mass functions. We propose and analyze a discontinuous Galerkin approximation for the Stokes problem. The finite element triangulation employed is not required to be conforming and we use discontinuous pressures and velocities. No additional unknown fields need to be introduced, but only suitable bilinear forms defined on the interfaces between the elements, involving the jumps of the velocity and the average of the pressure. We consider hp approximations using ℚk′–ℚk velocity-pressure pairs with k′ = k + 2, k + 1, k. Our methods show better stability properties than the corresponding conforming ones. We prove that our first two choices of velocity spaces ensure uniform divergence stability with respect to the mesh size h. Numerical results show that they are uniformly stable with respect to the local polynomial degree k, a property that has no analog in the conforming case. An explicit bound in k which is not sharp is also proven. Numerical results show that if equal order approximation is chosen for the velocity and pressure, no spurious pressure modes are present but the method is not uniformly stable either with respect to h or k. We derive a priori error estimates generalizing the abstract theory of mixed methods. Optimal error estimates in h are proven. As for discontinuous Galerkin methods for scalar diffusive problems, half of the power of k is lost for p and hp pproximations independently of the divergence stability. We present an algorithm that infers the model structure of a mixture of factor analysers using an efficient and deterministic variational approximation to full Bayesian integration over model parameters. This procedure can automatically determine the optimal number of components and the local dimensionality of each component (i.e. the number of factors in each factor analyser). Alternatively it can be used to infer posterior distributions over number of components and dimensionalities. Since all parameters are integrated out the method is not prone to overfitting. Using a stochastic procedure for adding components it is possible to perform the variational optimisation incrementally and to avoid local maxima. Results show that the method works very well in practice and correctly infers the number and dimensionality of nontrivial synthetic examples. By importance sampling from the variational approximation we show how to obtain unbiased estimates of the true evidence, the exact predictive density, and the KL divergence between the variational posterior and the true posterior, not only in this model but for variational approximations in general.