Performing statistical inference in high-dimension is an outstanding challenge. A major source of difficulty is the absence of precise information on the distribution of high-dimensional estimators. Here, we consider linear regression in the high-dimensional regime @math . In this context, we would like to perform inference on a high-dimensional parameters vector @math . Important progress has been achieved in computing confidence intervals for single coordinates @math . A key role in these new methods is played by a certain debiased estimator @math that is constructed from the Lasso. Earlier work establishes that, under suitable assumptions on the design matrix, the coordinates of @math are asymptotically Gaussian provided @math is @math -sparse with @math . The condition @math is stronger than the one for consistent estimation, namely @math . We study Gaussian designs with known or unknown population covariance. When the covariance is known, we prove that the debiased estimator is asymptotically Gaussian under the nearly optimal condition @math . Note that earlier work was limited to @math even for perfectly known covariance. The same conclusion holds if the population covariance is unknown but can be estimated sufficiently well, e.g. under the same sparsity conditions on the inverse covariance as assumed by earlier work. For intermediate regimes, we describe the trade-off between sparsity in the coefficients and in the inverse covariance of the design. We further discuss several applications of our results to high-dimensional inference. In particular, we propose a new estimator that is minimax optimal up to a factor @math for i.i.d. Gaussian designs.
The debiasing method was developed independently from several points of view @cite_37 @cite_0 @cite_43 @cite_29 @cite_9 . The present authors were motivated by the AMP analysis of the Lasso @cite_20 @cite_59 @cite_61 @cite_49 , and by the Gaussian limits that this analysis implies. In particular @cite_43 used those techniques to analyze standard Gaussian designs (i.e. the case @math ) in the asymptotic limit @math with @math , @math constant. In this limit, the debiased estimator was proven to be asymptotically Gaussian provided @math (for a universal constant @math ). This sparsity condition is even weaker than the one of Theorem (or Theorem ), but the result of @cite_43 only holds asymptotically. Also @cite_43 proved Gaussian convergence in a weaker sense than the one established here, implying coverage of the constructed confidence intervals only on average' over the coordinates @math .
@cite_61 @cite_37 @cite_29 @cite_9 @cite_0 @cite_43 @cite_59 @cite_49 @cite_20
2037228647 2127300249 2962726112 2123202508
We consider linear regression in the high-dimensional regime where the number of observations @math is smaller than the number of parameters @math . A very successful approach in this setting uses @math -penalized least squares (a.k.a. the Lasso) to search for a subset of @math parameters that best explain the data, while setting the other parameters to zero. Considerable amount of work has been devoted to characterizing the estimation and model selection problems within this approach. In this paper we consider instead the fundamental, but far less understood, question of . More precisely, we address the problem of computing p-values for single regression coefficients. On one hand, we develop a general upper bound on the minimax power of tests with a given significance level. On the other, we prove that this upper bound is (nearly) achievable through a practical procedure in the case of random design matrices with independent entries. Our approach is based on a debiasing of the Lasso estimator. The analysis builds on a rigorous characterization of the asymptotic distribution of the Lasso estimator and its debiased version. Our result holds for optimal sample size, i.e., when @math is at least on the order of @math . We generalize our approach to random design matrices with i.i.d. Gaussian rows @math . In this case we prove that a similar distributional characterization (termed standard distributional limit') holds for @math much larger than @math . Finally, we show that for optimal sample size, @math being at least of order @math , the standard distributional limit for general Gaussian designs can be derived from the replica heuristics in statistical physics. The problem of consistently estimating the sparsity pattern of a vector β* ∈ RP based on observations contaminated by noise arises in various contexts, including signal denoising, sparse approximation, compressed sensing, and model selection. We analyze the behavior of l1-constrained quadratic programming (QP), also referred to as the Lasso, for recovering the sparsity pattern. Our main result is to establish precise conditions on the problern dimension p, the number k of nonzero elements in β*, and the number of observations n that are necessary and sufficient for sparsity pattern recovery using the Lasso. We first analyze the case of observations made using deterministic design matrices and sub-Gaussian additive noise, and provide sufficient conditions for support recovery and l∞-error bounds, as well as results showing the necessity of incoherence and bounds on the minimum value. We then turn to the case of random designs, in which each row of the design is drawn from a N(0, Σ) ensemble. For a broad class of Gaussian ensembles satisfying mutual incoherence conditions, we compute explicit values of thresholds 0 0, if n > 2(θu + δ)k log(p - k), then the Lasso succeeds in recovering the sparsity pattern with probability converging to one for large problems, whereas for n < 2(θl - δ)k log(p - k), then the probability of successful recovery converges to zero. For the special case of the uniform Gaussian ensemble (Σ = I p×p), we show that θl = θu = 1, so that the precise threshold n = 2k log (p - k) is exactly determined. We consider the problem of fitting the parameters of a high-dimensional linear regression model. In the regime where the number of parameters p is comparable to or exceeds the sample size n, a successful approach uses an l1-penalized least squares estimator, known as Lasso. Unfortunately, unlike for linear estimators (e.g. ordinary least squares), no well-established method exists to compute confidence intervals or p-values on the basis of the Lasso estimator. Very recently, a line of work [8], [7], [13] has addressed this problem by constructing a debiased version of the Lasso estimator. We propose a special debiasing method that is well suited for random designs with sparse inverse covariance. Our approach improves over the state of the art in that it yields nearly optimal average testing power if sample size n asymptotically dominates s0(logp)2, with s0 being the sparsity level (number of non-zero coefficients). Earlier work achieved similar performances only for much larger sample size, namely it requires n to asymptotically dominates (s0 log p)2. We evaluate our method on synthetic data, and compare it with earlier proposals. We consider the problem of learning a coefficient vector xο ∈ RN from noisy linear observation y = Axo + ∈ Rn. In many contexts (ranging from model selection to image processing), it is desirable to construct a sparse estimator x. In this case, a popular approach consists in solving an l1-penalized least-squares problem known as the LASSO or basis pursuit denoising. For sequences of matrices A of increasing dimensions, with independent Gaussian entries, we prove that the normalized risk of the LASSO converges to a limit, and we obtain an explicit expression for this limit. Our result is the first rigorous derivation of an explicit formula for the asymptotic mean square error of the LASSO for random instances. The proof technique is based on the analysis of AMP, a recently developed efficient algorithm, that is inspired from graphical model ideas. Simulations on real data matrices suggest that our results can be relevant in a broad array of practical applications.