This paper considers the fundamental limit of compressed sensing for i.i.d. signal distributions and i.i.d. Gaussian measurement matrices. Its main contribution is a rigorous characterization of the asymptotic mutual information (MI) and minimum mean-square error (MMSE) in this setting. Under mild technical conditions, our results show that the limiting MI and MMSE are equal to the values predicted by the replica method from statistical physics. This resolves a well-known problem that has remained open for over a decade.
The replica method was developed originally to study mean-field approximations in spin glasses @cite_13 @cite_14 . It was first applied to linear estimation problems in the context of CDMA wireless communication @cite_45 @cite_23 @cite_28 , with subsequent work focusing on the compressed sensing directly @cite_26 @cite_38 @cite_11 @cite_43 @cite_25 @cite_39 .
@cite_38 @cite_14 @cite_26 @cite_28 @cite_39 @cite_43 @cite_45 @cite_23 @cite_13 @cite_25 @cite_11
2566505556 2550925785 2116394223 1967084746 2071108209
This book presents a unified approach to a rich and rapidly evolving research domain at the interface between statistical physics, theoretical computer science discrete mathematics, and coding information theory. It is accessible to graduate students and researchers without a specific training in any of these fields. The selected topics include spin glasses, error correcting codes, satisfiability, and are central to each field. The approach focuses on large random instances, adopting a common probabilistic formulation in terms of graphical models. It presents message passing algorithms like belief propagation and survey propagation, and their use in decoding and constraint satisfaction solving. It also explains analysis techniques like density evolution and the cavity method, and uses them to study phase transitions. Compressed sensing deals with the reconstruction of a high-dimensional signal from far fewer linear measurements, where the signal is known to admit a sparse representation in a certain linear space. The asymptotic scaling of the number of measurements needed for reconstruction as the dimension of the signal increases has been studied extensively. This work takes a fundamental perspective on the problem of inferring about individual elements of the sparse signal given the measurements, where the dimensions of the system become increasingly large. Using the replica method, the outcome of inferring about any fixed collection of signal elements is shown to be asymptotically decoupled, i.e., those elements become independent conditioned on the measurements. Furthermore, the problem of inferring about each signal element admits a single-letter characterization in the sense that the posterior distribution of the element, which is a sufficient statistic, becomes asymptotically identical to the posterior of inferring about the same element in scalar Gaussian noise. The result leads to simple characterization of all other elemental metrics of the compressed sensing problem, such as the mean squared error and the error probability for reconstructing the support set of the sparse signal. Finally, the single-letter characterization is rigorously justified in the special case of sparse measurement matrices where belief propagation becomes asymptotically optimal. We present a theory, based on statistical mechanics, to evaluate analytically the performance of uncoded, fully synchronous, randomly spread code-division multiple-access (CDMA) multiuser detectors with additive white Gaussian noise (AWGN) channel, under perfect power control, and in the large-system limit. Application of the replica method, a tool developed in the literature of statistical mechanics, allows us to derive analytical expressions for the bit-error rate, as well as the multiuser efficiency, of the individually optimum (IO) and jointly optimum (JO) multiuser detectors over the whole range of noise levels. The information-theoretic capacity of the randomly spread CDMA channel and the performance of decorrelating and linear minimum mean-square error (MMSE) detectors are also derived in the same replica formulation, thereby demonstrating validity of the statistical-mechanical approach. A new theory of the class of dilute magnetic alloys, called the spin glasses, is proposed which offers a simple explanation of the cusp found experimentally in the susceptibility. The argument is that because the interaction between the spins dissolved in the matrix oscillates in sign according to distance, there will be no mean ferro- or antiferromagnetism, but there will be a ground state with the spins aligned in definite directions, even if these directions appear to be at random. At the critical temperature the existence of these preferred directions affects the orientation of the spins, leading to a cusp in the susceptibility. This cusp is smoothed by an external field. Although the behaviour at low t needs a quantum mechanical treatment, it is interesting to complete the classical calculations down to t=0. Classically the susceptibility tends to a constant value at t=0, and the specific heat to a constant value. In recent work, two different methods have been used to characterize the fundamental limits of compressed sensing. On the one hand are rigorous bounds based on information-theoretic arguments or the analysis of specific algorithms. On the other hand are exact but heuristic predictions made using the replica method from statistical physics. In this paper, it is shown that, for certain problem settings, these bounds are in agreement, and thus provide a rigorous and accurate characterization of the compressed sensing problem. This characterization shows that the limits of sparse recovery can be quantified succinctly in terms of an effective signal-to-interference-plus-noise ratio, that depends on the number of measurements and the behavior of the sparse components themselves. Connections with the MMSE dimension by Wu and Verdu and minimax behavior of approximate message passing by are discussed.