Consider the standard linear regression model @math , where @math is an observation vector, @math is a design matrix, @math is the unknown regression vector, and @math is additive Gaussian noise. This paper studies the minimax rates of convergence for estimation of @math for @math -losses and in the @math -prediction loss, assuming that @math belongs to an @math -ball @math for some @math . We show that under suitable regularity conditions on the design matrix @math , the minimax error in @math -loss and @math -prediction loss scales as @math . In addition, we provide lower bounds on minimax risks in @math -norms, for all @math . Our proofs of the lower bounds are information-theoretic in nature, based on Fano's inequality and results on the metric entropy of the balls @math , whereas our proofs of the upper bounds are direct and constructive, involving direct analysis of least-squares over @math -balls. For the special case @math , a comparison with @math -risks achieved by computationally efficient @math -relaxations reveals that although such methods can achieve the minimax rates up to constant factors, they require slightly stronger assumptions on the design matrix @math than algorithms involving least-squares over the @math -ball.
Naturally, our work also has some connections to the vast body of work on @math -based methods for sparse estimation, particularly for the case of hard sparsity ( @math ). Based on our results, the rates that are achieved by @math -methods, such as the Lasso and the Dantzig selector, are minimax optimal for @math -loss, but require somewhat stronger conditions on the design matrix than an optimal'' algorithm, which is based on searching the @math -ball. We compare the conditions that we impose in our minimax analysis to various conditions imposed in the analysis of @math -based methods, including the restricted isometry property of Candes and Tao @cite_26 , the restricted eigenvalue condition imposed in Menshausen and Yu @cite_9 , the partial Riesz condition in Zhang and Huang @cite_24 and the restricted eigenvalue condition of @cite_28 . We find that optimal'' methods, which are based on minimizing least-squares directly over the @math -ball, can succeed for design matrices where @math -based methods are not known to work.
@cite_24 @cite_28 @cite_9 @cite_26
2295765951 2162312215 1486249389 2083945855
We consider the estimation of regression coefficients in a high-dimensional linear model. For regression coefficients in lr balls, we provide lower bounds for the minimax lq risk and minimax quantiles of the lq loss for all design matrices. Under an l0 sparsity condition on a target coefficient vector, we sharpen and unify existing oracle inequalities for the Lasso and Dantzig selector. We derive oracle inequalities for target coefficient vectors with many small elements and smaller threshold levels than the universal threshold. These oracle inequalities provide sufficient conditions on the design matrix for the rate minimaxity of the Lasso and Dantzig selector for the lq risk and loss in lr balls, 0≤ r≤ 1≤ q≤ ∞. By allowing q=∞, our risk bounds imply the variable selection consistency of threshold Lasso and Dantzig selectors. Methods based on l1-relaxation, such as basis pursuit and the Lasso, are very popular for sparse regression in high dimensions. The conditions for success of these methods are now well-understood: (1) exact recovery in the noiseless setting is possible if and only if the design matrix X satisfies the restricted nullspace property, and (2) the squared l2-error of a Lasso estimate decays at the minimax optimal rate k log p n, where k is the sparsity of the p-dimensional regression problem with additive Gaussian noise, whenever the design satisfies a restricted eigenvalue condition. The key issue is thus to determine when the design matrix X satisfies these desirable properties. Thus far, there have been numerous results showing that the restricted isometry property, which implies both the restricted nullspace and eigenvalue conditions, is satisfied when all entries of X are independent and identically distributed (i.i.d.), or the rows are unitary. This paper proves directly that the restricted nullspace and eigenvalue conditions hold with high probability for quite general classes of Gaussian matrices for which the predictors may be highly dependent, and hence restricted isometry conditions can be violated with high probability. In this way, our results extend the attractive theoretical guarantees on l1-relaxations to a much broader class of problems than the case of completely independent or unitary designs. Given @math noisy samples with @math dimensions, where @math , we show that the multi-step thresholding procedure based on the Lasso -- we call it the Thresholded Lasso , can accurately estimate a sparse vector @math in a linear model @math , where @math is a design matrix normalized to have column @math norm @math , and @math . We show that under the restricted eigenvalue (RE) condition (Bickel-Ritov-Tsybakov 09), it is possible to achieve the @math loss within a logarithmic factor of the ideal mean square error one would achieve with an oracle while selecting a sufficiently sparse model -- hence achieving sparse oracle inequalities ; the oracle would supply perfect information about which coordinates are non-zero and which are above the noise level. In some sense, the Thresholded Lasso recovers the choices that would have been made by the @math penalized least squares estimators, in that it selects a sufficiently sparse model without sacrificing the accuracy in estimating @math and in predicting @math . We also show for the Gauss-Dantzig selector (Cand e s-Tao 07), if @math obeys a uniform uncertainty principle and if the true parameter is sufficiently sparse, one will achieve the sparse oracle inequalities as above, while allowing at most @math irrelevant variables in the model in the worst case, where @math is the smallest integer such that for @math , @math . Our simulation results on the Thresholded Lasso match our theoretical analysis excellently. Abstract We investigate the recovery of almost s -sparse vectors x ∈ C N from undersampled and inaccurate data y = A x + e ∈ C m by means of minimizing ‖ z ‖ 1 subject to the equality constraints A z = y . If m ≍ s ln ( N s ) and if Gaussian random matrices A ∈ R m × N are used, this equality-constrained l 1 -minimization is known to be stable with respect to sparsity defects and robust with respect to measurement errors. If m ≍ s ln ( N s ) and if Weibull random matrices are used, we prove here that the equality-constrained l 1 -minimization remains stable and robust. The arguments are based on two key ingredients, namely the robust null space property and the quotient property. The robust null space property relies on a variant of the classical restricted isometry property where the inner norm is replaced by the l 1 -norm and the outer norm is replaced by a norm comparable to the l 2 -norm. For the l 1 -minimization subject to inequality constraints, this yields stability and robustness results that are also valid when considering sparsity relative to a redundant dictionary. As for the quotient property, it relies on lower estimates for the tail probability of sums of independent Weibull random variables.