This article is an extended version of previous work of the authors [40, 41] on low-rank matrix estimation in the presence of constraints on the factors into which the matrix is factorized. Low-rank matrix factorization is one of the basic methods used in data analysis for unsupervised learning of relevant features and other types of dimensionality reduction. We present a framework to study the constrained low-rank matrix estimation for a general prior on the factors, and a general output channel through which the matrix is observed. We draw a paralel with the study of vector-spin glass models - presenting a unifying way to study a number of problems considered previously in separate statistical physics works. We present a number of applications for the problem in data analysis. We derive in detail a general form of the low-rank approximate message passing (Low- RAMP) algorithm, that is known in statistical physics as the TAP equations. We thus unify the derivation of the TAP equations for models as different as the Sherrington-Kirkpatrick model, the restricted Boltzmann machine, the Hopfield model or vector (xy, Heisenberg and other) spin glasses. The state evolution of the Low-RAMP algorithm is also derived, and is equivalent to the replica symmetric solution for the large class of vector-spin glass models. In the section devoted to result we study in detail phase diagrams and phase transitions for the Bayes-optimal inference in low-rank matrix estimation. We present a typology of phase transitions and their relation to performance of algorithms such as the Low-RAMP or commonly used spectral methods.
The present paper is built upon two previous shorter papers by the same authors @cite_33 @cite_54 . it focuses on the study of the general type of models described by probability measures ) and ). On the one hand, these represent Boltzmann measures of vectorial-spin systems on fully connected symmetric or bipartite graphs. Examples of previously studied physical models that are special cases of the setting considered here would be the Sherrington-Kirkpatrick model @cite_22 , the Hopfield model @cite_40 @cite_8 , the inference (not learning) in the restricted Boltzmann machine @cite_29 @cite_8 @cite_6 . Our work hence provides a unified replica symmetric solution and TAP equations for generic class of prior distributions and Hamiltonians.
@cite_33 @cite_22 @cite_8 @cite_54 @cite_29 @cite_6 @cite_40
2165225968 2018836315 1884565747 2243462087
We present a distribution model for binary vectors, called the influence combination model and show how this model can be used as the basis for unsupervised learning algorithms for feature selection. The model can be represented by a particular type of Boltzmann machine with a bipartite graph structure that we call the combination machine. This machine is closely related to the Harmonium model defined by Smolensky. In the first part of the paper we analyze properties of this distribution representation scheme. We show that arbitrary distributions of binary vectors can be approximated by the combination model. We show how the weight vectors in the model can be interpreted as high order correlation patterns among the input bits, and how the combination machine can be used as a mechanism for detecting these patterns. We compare the combination model with the mixture model and with principle component analysis. In the second part of the paper we present two algorithms for learning the combination model from examples. The first learning algorithm is the standard gradient ascent heuristic for computing maximum likelihood estimates for the parameters of the model. Here we give a closed form for this gradient that is significantly easier to compute than the corresponding gradient for the general Boltzmann machine. The second learning algorithm is a greedy method that creates the hidden units and computes their weights one at a time. This method is a variant of projection pursuit density estimation. In the third part of the paper we give experimental results for these learning methods on synthetic data and on natural data of handwritten digit images. We consider homogeneous factor models on uniformly sparse graph sequences converging locally to a (unimodular) random tree T, and study the existence of the free energy density ϕ, the limit of the log-partition function divided by the number of vertices n as n tends to infinity. We provide a new interpolation scheme and use it to prove existence of, and to explicitly compute, the quantity ϕ subject to uniqueness of a relevant Gibbs measure for the factor model on T. By way of example we compute ϕ for the independent set (or hard-core) model at low fugacity, for the ferromagnetic Ising model at all parameter values, and for the ferromagnetic Potts model with both weak enough and strong enough interactions. Even beyond uniqueness regimes our interpolation provides useful explicit bounds on ϕ. In the regimes in which we establish existence of the limit, we show that it coincides with the Bethe free energy functional evaluated at a suitable fixed point of the belief propagation (Bethe) recursions on T. In the special case that T has a Galton–Watson law, this formula coincides with the nonrigorous “Bethe prediction” obtained by statistical physicists using the “replica” or “cavity” methods. Thus our work is a rigorous generalization of these heuristic calculations to the broader class of sparse graph sequences converging locally to trees. We also provide a variational characterization for the Bethe prediction in this general setting, which is of independent interest. We consider models given by Hamiltonians of the form View the MathML sourceH(I,f,p,q,t;e)=h(I)+?j=1n±(12pj2+Vj(qj))+eQ(I,f,p,q,t;e) Turn MathJax on where I?I?Rd,f?TdI?I?Rd,f?Td, p,q?Rnp,q?Rn, t?T1t?T1. These are higher dimensional analogues, both in the center and hyperbolic directions, of the models studied in , and and are usually called “a-priori unstable Hamiltonian systems”. All these models present the large gap problem. We show that, for 0<e«10<e«1, under regularity and explicit non-degeneracy conditions on the model, there are orbits whose action variables I perform rather arbitrary excursions in a domain of size O(1)O(1). This domain includes resonance lines and, hence, large gaps among d-dimensional KAM tori. This phenomenon is known as Arnold diffusion. The method of proof follows closely the strategy of and . The main new phenomenon that appears when the dimension d of the center directions is larger than one is the existence of multiple resonances in the space of actions I?I?RdI?I?Rd. We show that, since these multiple resonances happen in sets of codimension greater than one in the space of actions I, they can be contoured. This corresponds to the mechanism called diffusion across resonances in the Physics literature. The present paper, however, differs substantially from and . On the technical details of the proofs, we have taken advantage of the theory of the scattering map developed in [31]—notably the symplectic properties—which were not available when the above papers were written. We have analyzed the conditions imposed on the resonances in more detail. More precisely, we have found that there is a simple condition on the Melnikov potential which allows us to conclude that the resonances are crossed. In particular, this condition does not depend on the resonances. So that the results are new even when applied to the models in and We develop a novel modeling framework for Boltzmann machines, augmenting each hidden unit with a latent transformation assignment variable which describes the selection of the transformed view of the canonical connection weights associated with the unit. This enables the inferences of the model to transform in response to transformed input data in a stable and predictable way, and avoids learning multiple features differing only with respect to the set of transformations. Extending prior work on translation equivariant (convolutional) models, we develop translation and rotation equivariant restricted Boltzmann machines (RBMs) and deep belief nets (DBNs), and demonstrate their effectiveness in learning frequently occurring statistical structure from artificial and natural images.