We study the estimation error of constrained M-estimators, and derive explicit upper bounds on the expected estimation error determined by the Gaussian width of the constraint set. Both of the cases where the true parameter is on the boundary of the constraint set (matched constraint), and where the true parameter is strictly in the constraint set (mismatched constraint) are considered. For both cases, we derive novel universal estimation error bounds for regression in a generalized linear model with the canonical link function. Our error bound for the mismatched constraint case is minimax optimal in terms of its dependence on the sample size, for Gaussian linear regression by the Lasso.
In @cite_20 @cite_7 , the authors derived sharp estimation error bounds for regression in the linear model by constrained least squares (LS) estimators. The analysis in @cite_0 provides a minimax estimation error bound for the same setting . There are some related works on learning a function in a function class @cite_2 @cite_6 . When the function class is linearly parametrized by vectors in @math , and the function corresponding to @math is in the function class, the @math -estimation error in the function class may be translated into the @math -estimation error with respect to @math . A common limitation of @cite_2 @cite_6 @cite_7 @cite_20 @cite_0 is that the results are not extendable to general non-linear statistical models.
@cite_7 @cite_6 @cite_0 @cite_2 @cite_20
2895165641 2963403872 2059867647 1896002532
We study the problem of estimating a set of @math linear queries with respect to some unknown distribution @math over a domain @math based on a sensitive data set of @math individuals under the constraint of local differential privacy. This problem subsumes a wide range of estimation tasks, e.g., distribution estimation and @math -dimensional mean estimation. We provide new algorithms for both the offline (non-adaptive) and adaptive versions of this problem. In the offline setting, the set of queries are fixed before the algorithm starts. In the regime where @math , our algorithms attain @math estimation error that is independent of @math , and is tight up to a factor of @math . For the special case of distribution estimation, we show that projecting the output estimate of an algorithm due to [ 2018] on the probability simplex yields an @math error that depends only sub-logarithmically on @math in the regime where @math . These results show the possibility of accurate estimation of linear queries in the high-dimensional settings under the @math error criterion. In the adaptive setting, the queries are generated over @math rounds; one query at a time. In each round, a query can be chosen adaptively based on all the history of previous queries and answers. We give an algorithm for this problem with optimal @math estimation error (worst error in the estimated values for the queries w.r.t. the data distribution). Our bound matches a lower bound on the @math error for the offline version of this problem [ 2013]. Author(s): Plan, Y; Vershynin, R; Yudovina, E | Abstract: Consider measuring an n-dimensional vector x through the inner product with several measurement vectors, a_1, a_2, ..., a_m. It is common in both signal processing and statistics to assume the linear response model y_i = + e_i, where e_i is a noise term. However, in practice the precise relationship between the signal x and the observations y_i may not follow the linear model, and in some cases it may not even be known. To address this challenge, in this paper we propose a general model where it is only assumed that each observation y_i may depend on a_i only through . We do not assume that the dependence is known. This is a form of the semiparametric single index model, and it includes the linear model as well as many forms of the generalized linear model as special cases. We further assume that the signal x has some structure, and we formulate this as a general assumption that x belongs to some known (but arbitrary) feasible set K. We carefully detail the benefit of using the signal structure to improve estimation. The theory is based on the mean width of K, a geometric parameter which can be used to understand its effective dimension in estimation problems. We determine a simple, efficient two-step procedure for estimating the signal based on this model -- a linear estimation followed by metric projection onto K. We give general conditions under which the estimator is minimax optimal up to a constant. This leads to the intriguing conclusion that in the high noise regime, an unknown non-linearity in the observations does not significantly reduce one's ability to determine the signal, even when the non-linearity may be non-invertible. Our results may be specialized to understand the effect of non-linearities in compressed sensing. We give near-optimal space bounds in the streaming model for linear algebra problems that include estimation of matrix products, linear regression, low-rank approximation, and approximation of matrix rank. In the streaming model, sketches of input matrices are maintained under updates of matrix entries; we prove results for turnstile updates, given in an arbitrary order. We give the first lower bounds known for the space needed by the sketches, for a given estimation error e. We sharpen prior upper bounds, with respect to combinations of space, failure probability, and number of passes. The sketch we use for matrix A is simply STA, where S is a sign matrix. Our results include the following upper and lower bounds on the bits of space needed for 1-pass algorithms. Here A is an n x d matrix, B is an n x d' matrix, and c := d+d'. These results are given for fixed failure probability; for failure probability δ>0, the upper bounds require a factor of log(1 δ) more space. We assume the inputs have integer entries specified by O(log(nc)) bits, or O(log(nd)) bits. (Matrix Product) Output matrix C with F(ATB-C) ≤ e F(A) F(B). We show that Θ(ce-2log(nc)) space is needed. (Linear Regression) For d'=1, so that B is a vector b, find x so that Ax-b ≤ (1+e) minx' ∈ Reald Ax'-b. We show that Θ(d2e-1 log(nd)) space is needed. (Rank-k Approximation) Find matrix tAk of rank no more than k, so that F(A-tAk) ≤ (1+e) F A-Ak , where Ak is the best rank-k approximation to A. Our lower bound is Ω(ke-1(n+d)log(nd)) space, and we give a one-pass algorithm matching this when A is given row-wise or column-wise. For general updates, we give a one-pass algorithm needing [O(ke-2(n + d e2)log(nd))] space. We also give upper and lower bounds for algorithms using multiple passes, and a sketching analog of the CUR decomposition. In many problems such as linear and nonlinear regressions, parameter and state estimation of dynamic systems, state-space and time series prediction, interpolation, smoothing and functions approximation, one has to evaluate some unknown variable using available data. The data are always associated with some uncertainty and it is necessary to evaluate how this uncertainty affects the estimated variables. Typically, the problem is approached assuming a probabilistic description of uncertainty and applying statistical estimation theory. An interesting alternative approach, referred to as set membership or unknown but bounded (UBB) error description, has been investigated since the late 1960s. In this approach, uncertainty is described by an additive noise which is known only to have given integral (typically l2 of l1) or componentwise (l∞) bounds. In this paper we review the main results of this theory, with special attention to the most recent advances obtained in the case of componentwise bounds.