Iterative thresholding algorithms are well-suited for high-dimensional problems in sparse recovery and compressive sensing. The performance of this class of algorithms depends heavily on the tuning of certain threshold parameters. In particular, both the final reconstruction error and the convergence rate of the algorithm crucially rely on how the threshold parameter is set at each step of the algorithm. In this paper, we propose a parameter-free approximate message passing (AMP) algorithm that sets the threshold parameter at each iteration in a fully automatic way without either having an information about the signal to be reconstructed or needing any tuning from the user. We show that the proposed method attains both the minimum reconstruction error and the highest convergence rate. Our method is based on applying the Stein unbiased risk estimate (SURE) along with a modified gradient descent to find the optimal threshold in each iteration. Motivated by the connections between AMP and LASSO, it could be employed to find the solution of the LASSO for the optimal regularization parameter. To the best of our knowledge, this is the first work concerning parameter tuning that obtains the fastest convergence rate with theoretical guarantees.
Optimal tuning of parameters to obtain the smallest final reconstruction error has been the focus of major research in CS, machine learning, and statistics. The methods considered in the literature fall into the following three categories: [(i)] The first approach is based on obtaining an upper bound for the reconstruction error and setting the parameters to obtain the smallest upper bound. For many of the algorithms proposed in the literature, there exists a theoretical analysis based on certain properties of the matrix, such as RIP @cite_24 @cite_6 , Coherence @cite_27 , and RSC @cite_16 . These analyses can potentially provide a simple approach for tuning parameters. However, they suffer from two issues: (i) Inaccuracy of the upper bounds derived for the risk of the final estimates usually lead to pessimistic parameter choices that are not useful for practical purposes. (ii) The requirement of an upper bound for the sparsity level @cite_9 @cite_12 , which is often not available in practice.
@cite_9 @cite_6 @cite_24 @cite_27 @cite_16 @cite_12
1834085549 1977252496 2071099763 2127300249
High-dimensional binary classification tasks, e.g. the classification of microarray samples into normal and cancer tissues, usually involve a tuning parameter adjusting the complexity of the applied method to the examined data set. By reporting the performance of the best tuning parameter value only, over-optimistic prediction errors are published. The contribution of this paper is two-fold. Firstly, we develop a new method for tuning bias correction which can be motivated by decision theoretic considerations. The method is based on the decomposition of the unconditional error rate involving the tuning procedure. Our corrected error estimator can be written as a weighted mean of the errors obtained using the different tuning parameter values. It can be interpreted as a smooth version of nested cross-validation (NCV) which is the standard approach for avoiding tuning bias. In contrast to NCV, the weighting scheme of our method guarantees intuitive bounds for the corrected error. Secondly, we suggest to use bias correction methods also to address the bias resulting from the optimal choice of the classification method among several competitors. This method selection bias is particularly relevant to prediction problems in high-dimensional data. In the absence of standards, it is common practice to try several methods successively, which can lead to an optimistic bias similar to the tuning bias. We demonstrate the performance of our method to address both types of bias based on microarray data sets and compare it to existing methods. This study confirms that our approach yields estimates competitive to NCV at a much lower computational price. We study the notion of compressed sensing (CS) as put forward by Donoho, Candes, Tao and others. The notion proposes a signal or image, unknown but supposed to be compressible by a known transform, (e.g. wavelet or Fourier), can be subjected to fewer measurements than the nominal number of data points, and yet be accurately reconstructed. The samples are nonadaptive and measure 'random' linear combinations of the transform coefficients. Approximate reconstruction is obtained by solving for the transform coefficients consistent with measured data and having the smallest possible l1 norm.We present initial 'proof-of-concept' examples in the favorable case where the vast majority of the transform coefficients are zero. We continue with a series of numerical experiments, for the setting of lp-sparsity, in which the object has all coefficients nonzero, but the coefficients obey an lp bound, for some p ∈ (0, 1]. The reconstruction errors obey the inequalities paralleling the theory, seemingly with well-behaved constants.We report that several workable families of 'random' linear combinations all behave equivalently, including random spherical, random signs, partial Fourier and partial Hadamard.We next consider how these ideas can be used to model problems in spectroscopy and image processing, and in synthetic examples see that the reconstructions from CS are often visually "noisy". To suppress this noise we postprocess using translation-invariant denoising, and find the visual appearance considerably improved.We also consider a multiscale deployment of compressed sensing, in which various scales are segregated and CS applied separately to each; this gives much better quality reconstructions than a literal deployment of the CS methodology.These results show that, when appropriately deployed in a favorable setting, the CS framework is able to save significantly over traditional sampling, and there are many useful extensions of the basic idea. In computed tomography there are different situations where reconstruction has to be performed with limited raw data. In the past few years it has been shown that algorithms which are based on compressed sensing theory are able to handle incomplete datasets quite well. As a cost function these algorithms use the l1-norm of the image after it has been transformed by a sparsifying transformation. This yields to an inequality-constrained convex optimization problem. Due to the large size of the optimization problem some heuristic optimization algorithms have been proposed in the past few years. The most popular way is optimizing the raw data and sparsity cost functions separately in an alternating manner. In this paper we will follow this strategy and present a new method to adapt these optimization steps. Compared to existing methods which perform similarly, the proposed method needs no a priori knowledge about the raw data consistency. It is ensured that the algorithm converges to the lowest possible value of the raw data cost function, while holding the sparsity constraint at a low value. This is achieved by transferring the step-size determination of both optimization procedures into the raw data domain, where they are adapted to each other. To evaluate the algorithm, we process measured clinical datasets. To cover a wide field of possible applications, we focus on the problems of angular undersampling, data lost due to met al implants, limited view angle tomography and interior tomography. In all cases the presented method reaches convergence within less than 25 iteration steps, while using a constant set of algorithm control parameters. The image artifacts caused by incomplete raw data are mostly removed without introducing new effects like staircasing. All scenarios are compared to an existing implementation of the ASD-POCS algorithm, which realizes the step-size adaption in a different way. Additional prior information as proposed by the PICCS algorithm can be incorporated easily into the optimization process. The problem of consistently estimating the sparsity pattern of a vector β* ∈ RP based on observations contaminated by noise arises in various contexts, including signal denoising, sparse approximation, compressed sensing, and model selection. We analyze the behavior of l1-constrained quadratic programming (QP), also referred to as the Lasso, for recovering the sparsity pattern. Our main result is to establish precise conditions on the problern dimension p, the number k of nonzero elements in β*, and the number of observations n that are necessary and sufficient for sparsity pattern recovery using the Lasso. We first analyze the case of observations made using deterministic design matrices and sub-Gaussian additive noise, and provide sufficient conditions for support recovery and l∞-error bounds, as well as results showing the necessity of incoherence and bounds on the minimum value. We then turn to the case of random designs, in which each row of the design is drawn from a N(0, Σ) ensemble. For a broad class of Gaussian ensembles satisfying mutual incoherence conditions, we compute explicit values of thresholds 0 0, if n > 2(θu + δ)k log(p - k), then the Lasso succeeds in recovering the sparsity pattern with probability converging to one for large problems, whereas for n < 2(θl - δ)k log(p - k), then the probability of successful recovery converges to zero. For the special case of the uniform Gaussian ensemble (Σ = I p×p), we show that θl = θu = 1, so that the precise threshold n = 2k log (p - k) is exactly determined.