Approximate Survey Propagation for Statistical Inference <s> II. PROPERTIES OF APPROXIMATE MESSAGE PASSING <s> We consider an Ising model in which the spins are coupled by infinite-ranged random interactions independently distributed with a Gaussian probability density. Both "spinglass" and ferromagnetic phases occur. The competition between the phases and the type of order present in each are studied. <s> BIB001 </s> Approximate Survey Propagation for Statistical Inference <s> II. PROPERTIES OF APPROXIMATE MESSAGE PASSING <s> Abstract The Sherrmgton-Kirkpatrick model of a spin glass is solved by a mean field technique which is probably exact in the limit of infinite range interactions. At and above T c the solution is identical to that obtained by Sherrington and Kirkpatrick (1975) using the n → O replica method, but below T c the new result exhibits several differences and remains physical down to T = 0. <s> BIB002 </s> Approximate Survey Propagation for Statistical Inference <s> II. PROPERTIES OF APPROXIMATE MESSAGE PASSING <s> This book contains a detailed and self-contained presentation of the replica theory of infinite range spin glasses. The authors also explain recent theoretical developments, paying particular attention to new applications in the study of optimization theory and neural networks. About two-thirds of the book are a collection of the most interesting and pedagogical articles on the subject. <s> BIB003 </s> Approximate Survey Propagation for Statistical Inference <s> II. PROPERTIES OF APPROXIMATE MESSAGE PASSING <s> Motivated by recent progress in using restricted Boltzmann machines as preprocess-ing algorithms for deep neural network, we revisit the mean-field equations (belief-propagation and TAP equations) in the best understood such machine, namely the Hopfield model of neural networks, and we explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polariza-tions of neurons. In the "retrieval phase" where neurons polarize in the direction of one memorized pattern, we point out a major difference between the belief propagation and TAP equations : the set of belief propagation equations depends on the pattern which is retrieved, while one can use a unique set of TAP equations. This makes the latter method much better suited for applications in the learning process of restricted Boltzmann machines. In the case where the patterns memorized in the Hopfield model are not independent, but are correlated through a combinatorial structure, we show that the TAP equations have to be modified. This modification can be seen either as an alteration of the reaction term in TAP equations, or, more interestingly, as the consequence of message passing on a graphical model with several hidden layers, where the number of hidden layers depends on the depth of the correlations in the memorized patterns. This layered structure is actually necessary when one deals with more general restricted Boltzmann machines. <s> BIB004 </s> Approximate Survey Propagation for Statistical Inference <s> II. PROPERTIES OF APPROXIMATE MESSAGE PASSING <s> In this work, we consider compressed sensing reconstruction from M measurements of K-sparse structured signals which do not possess a writable correlation model. Assuming that a generative statistical model, such as a Boltzmann machine, can be trained in an unsupervised manner on example signals, we demonstrate how this signal model can be used within a Bayesian framework of signal reconstruction. By deriving a message-passing inference for general distribution restricted Boltzmann machines, we are able to integrate these inferred signal models into approximate message passing for compressed sensing reconstruction. Finally, we show for the MNIST dataset that this approach can be very effective, even for M <; K. <s> BIB005 </s> Approximate Survey Propagation for Statistical Inference <s> II. PROPERTIES OF APPROXIMATE MESSAGE PASSING <s> Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine--learning tasks. Restricted Boltzmann Machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a compositional phase. Evidence is provided by the replica analysis of an adequate statistical ensemble of random RBMs and by RBM trained on the handwritten digits dataset MNIST. <s> BIB006 </s> Approximate Survey Propagation for Statistical Inference <s> II. PROPERTIES OF APPROXIMATE MESSAGE PASSING <s> This article is an extended version of previous work of the authors [40, 41] on low-rank matrix estimation in the presence of constraints on the factors into which the matrix is factorized. Low-rank matrix factorization is one of the basic methods used in data analysis for unsupervised learning of relevant features and other types of dimensionality reduction. We present a framework to study the constrained low-rank matrix estimation for a general prior on the factors, and a general output channel through which the matrix is observed. We draw a paralel with the study of vector-spin glass models - presenting a unifying way to study a number of problems considered previously in separate statistical physics works. We present a number of applications for the problem in data analysis. We derive in detail a general form of the low-rank approximate message passing (Low- RAMP) algorithm, that is known in statistical physics as the TAP equations. We thus unify the derivation of the TAP equations for models as different as the Sherrington-Kirkpatrick model, the restricted Boltzmann machine, the Hopfield model or vector (xy, Heisenberg and other) spin glasses. The state evolution of the Low-RAMP algorithm is also derived, and is equivalent to the replica symmetric solution for the large class of vector-spin glass models. In the section devoted to result we study in detail phase diagrams and phase transitions for the Bayes-optimal inference in low-rank matrix estimation. We present a typology of phase transitions and their relation to performance of algorithms such as the Low-RAMP or commonly used spectral methods. <s> BIB007
A. Reminder of AMP and its state evolution In this section we recall the standard Approximate Message Passing (AMP) algorithm. Within the context of low-rank matrix estimation, the AMP equations are referred as Low-RAMP and are discussed extensively in BIB007 . In the physics literature, the Low-RAMP would be equivalent to the TAP equations BIB002 (with corrected iteration indices) for a model of vectorial spins with local magnetic fields and general kind of two-body interactions. In this sense, the Low-RAMP equations encompass as a special case the original TAP equations for the SherringtonKirkpatrick model BIB001 , for the Hopfield model BIB003 BIB004 and for the restricted Boltzmann machine BIB004  BIB005 BIB006 .