This paper considers the fundamental limit of compressed sensing for i.i.d. signal distributions and i.i.d. Gaussian measurement matrices. Its main contribution is a rigorous characterization of the asymptotic mutual information (MI) and minimum mean-square error (MMSE) in this setting. Under mild technical conditions, our results show that the limiting MI and MMSE are equal to the values predicted by the replica method from statistical physics. This resolves a well-known problem that has remained open for over a decade.
Recent work by Huleihel and Merhav @cite_18 addresses the validity of the replica MMSE directly in the case of Gaussian mixture models, using tools from statistical physics and random matrix theory @cite_5 @cite_30 .
@cite_30 @cite_5 @cite_18
2116394223 2089912168 2168671066 2094248329
We present a theory, based on statistical mechanics, to evaluate analytically the performance of uncoded, fully synchronous, randomly spread code-division multiple-access (CDMA) multiuser detectors with additive white Gaussian noise (AWGN) channel, under perfect power control, and in the large-system limit. Application of the replica method, a tool developed in the literature of statistical mechanics, allows us to derive analytical expressions for the bit-error rate, as well as the multiuser efficiency, of the individually optimum (IO) and jointly optimum (JO) multiuser detectors over the whole range of noise levels. The information-theoretic capacity of the randomly spread CDMA channel and the performance of decorrelating and linear minimum mean-square error (MMSE) detectors are also derived in the same replica formulation, thereby demonstrating validity of the statistical-mechanical approach. Mixture modelling involves explaining some observed evidence using a combination of probability distributions. The crux of the problem is the inference of an optimal number of mixture components and their corresponding parameters. This paper discusses unsupervised learning of mixture models using the Bayesian Minimum Message Length (MML) criterion. To demonstrate the effectiveness of search and inference of mixture parameters using the proposed approach, we select two key probability distributions, each handling fundamentally different types of data: the multivariate Gaussian distribution to address mixture modelling of data distributed in Euclidean space, and the multivariate von Mises-Fisher (vMF) distribution to address mixture modelling of directional data distributed on a unit hypersphere. The key contributions of this paper, in addition to the general search and inference methodology, include the derivation of MML expressions for encoding the data using multivariate Gaussian and von Mises-Fisher distributions, and the analytical derivation of the MML estimates of the parameters of the two distributions. Our approach is tested on simulated and real world data sets. For instance, we infer vMF mixtures that concisely explain experimentally determined three-dimensional protein conformations, providing an effective null model description of protein structures that is central to many inference problems in structural bioinformatics. The experimental results demonstrate that the performance of our proposed search and inference method along with the encoding schemes improve on the state of the art mixture modelling techniques. In continuation to a recent work on the statistical-mechanical analysis of minimum mean square error (MMSE) estimation in Gaussian noise via its relation to the mutual information (the I-MMSE relation), here we propose a simple and more direct relationship between optimum estimation and certain information measures (e.g., the information density and the Fisher information), which can be viewed as partition functions and hence are amenable to analysis using statistical-mechanical techniques. The proposed approach has several advantages, most notably, its applicability to general sources and channels, as opposed to the I-MMSE relation and its variants which hold only for certain classes of channels (e.g., additive white Gaussian noise channels). We then demonstrate the derivation of the conditional mean estimator and the MMSE in a few examples. Two of these examples turn out to be generalizable to a fairly wide class of sources and channels. For this class, the proposed approach is shown to yield an approximate conditional mean estimator and an MMSE formula that has the flavor of a single-letter expression. We also show how our approach can easily be generalized to situations of mismatched estimation. A hierarchical Bayes Gaussian Mixture Model (HBGMM) with adaptive sparsity is proposed.We impose our hierarchical Bayes prior on the entries of precision matrices in GMM to encourage sparsity.Our adaptive sparsity can reduce the dimensionality of the parameters in GMM to be estimated.Our HBGMM method does not involve any hyperparameters to be estimated, i.e., it is hyperparameter-free. Gaussian Mixture Model (GMM) has been widely used in statistics for its great flexibility. However, parameter estimation for GMM with high dimensionality is a challenge because of the large number of parameters and the lack of observation data. In this paper, we propose an effective method named hierarchical Bayes based Adaptive Sparsity in Gaussian Mixture Model (ASGMM) to estimate the parameters in a GMM by incorporating a two-layer hierarchical Bayes based adaptive sparsity prior. The prior we impose on the precision matrices can encourage sparsity and hence reduce the dimensionality of the parameters to be estimated. In contrast to the l 1 -norm penalty or Laplace prior, our approach does not involve any hyperparameters that must be tuned, and the sparsity adapts to the observation data. The proposed method is achieved by three steps: first, we formulate an adaptive hierarchical Bayes model of the precision matrices in the GMM with a Jeffrey's noninformative hyperprior, which expresses scale-invariance and, more importantly, is hyperparameter-free and unbiased. Second, we perform a Cholesky decomposition on the precision matrices to impose the positive definite property. Finally, we exploit the expectation maximization (EM) algorithm to obtain the final estimated parameters in the GMM. Experimental results on synthetic and real-world datasets demonstrate that ASGMM cannot only adapt the sparsity of high-dimensional data with small estimated error, but also achieve better clustering performance comparing with several classical methods.