We consider the mixed regression problem with two components, under adversarial and stochastic noise. We give a convex optimization formulation that provably recovers the true solution, and provide upper bounds on the recovery errors for both arbitrary noise and stochastic noise settings. We also give matching minimax lower bounds (up to log factors), showing that under certain assumptions, our algorithm is information-theoretically optimal. Our results represent the first tractable algorithm guaranteeing successful recovery with tight bounds on recovery errors and sample complexity.
One exception is the recent work in @cite_13 , which considers mixed regression in the noiseless setting, where they propose an alternating minimization approach initialized by a grid search and show that it recovers the regressors in the noiseless case with a sample complexity of @math . Extension to the noisy setting is very recently considered in @cite_1 . Focusing on the stochastic noise setting and the high SNR regime (i.e., when @math ; cf. ), they show that the EM algorithm with good initialization achieves the error bound @math . Another notable exception is the work in @cite_31 . There, EM is adapted to the high-dimensional sparse regression setting, where the regressors are known to be sparse. The authors use EM to solve a penalized (for sparsity) likelihood function. A generalized EM approach achieves support-recovery, though once restricted to that support where the problem becomes a standard mixed regression problem, only convergence to a local optimum can be guaranteed.
@cite_1 @cite_31 @cite_13
1664169458 2950162031 2162312215 2114147096
We consider high dimensional sparse regression, and develop strategies able to deal with arbitrary -- possibly, severe or coordinated -- errors in the covariance matrix @math . These may come from corrupted data, persistent experimental errors, or malicious respondents in surveys recommender systems, etc. Such non-stochastic error-in-variables problems are notoriously difficult to treat, and as we demonstrate, the problem is particularly pronounced in high-dimensional settings where the primary goal is support recovery of the sparse regressor. We develop algorithms for support recovery in sparse regression, when some number @math out of @math total covariate response pairs are arbitrarily (possibly maliciously) corrupted . We are interested in understanding how many outliers, @math , we can tolerate, while identifying the correct support. To the best of our knowledge, neither standard outlier rejection techniques, nor recently developed robust regression algorithms (that focus only on corrupted response variables), nor recent algorithms for dealing with stochastic noise or erasures, can provide guarantees on support recovery. Perhaps surprisingly, we also show that the natural brute force algorithm that searches over all subsets of @math covariate response pairs, and all subsets of possible support coordinates in order to minimize regression error, is remarkably poor, unable to correctly identify the support with even @math corrupted points, where @math is the sparsity. This is true even in the basic setting we consider, where all authentic measurements and noise are independent and sub-Gaussian. In this setting, we provide a simple algorithm -- no more computationally taxing than OMP -- that gives stronger performance guarantees, recovering the support with up to @math corrupted points, where @math is the dimension of the signal to be recovered. We investigate the clustering performances of the relaxed @math means in the setting of sub-Gaussian Mixture Model (sGMM) and Stochastic Block Model (SBM). After identifying the appropriate signal-to-noise ratio (SNR), we prove that the misclassification error decay exponentially fast with respect to this SNR. These partial recovery bounds for the relaxed @math means improve upon results currently known in the sGMM setting. In the SBM setting, applying the relaxed @math means SDP allows to handle general connection probabilities whereas other SDPs investigated in the literature are restricted to the assortative case (where within group probabilities are larger than between group probabilities). Again, this partial recovery bound complements the state-of-the-art results. All together, these results put forward the versatility of the relaxed @math means. Methods based on l1-relaxation, such as basis pursuit and the Lasso, are very popular for sparse regression in high dimensions. The conditions for success of these methods are now well-understood: (1) exact recovery in the noiseless setting is possible if and only if the design matrix X satisfies the restricted nullspace property, and (2) the squared l2-error of a Lasso estimate decays at the minimax optimal rate k log p n, where k is the sparsity of the p-dimensional regression problem with additive Gaussian noise, whenever the design satisfies a restricted eigenvalue condition. The key issue is thus to determine when the design matrix X satisfies these desirable properties. Thus far, there have been numerous results showing that the restricted isometry property, which implies both the restricted nullspace and eigenvalue conditions, is satisfied when all entries of X are independent and identically distributed (i.i.d.), or the rows are unitary. This paper proves directly that the restricted nullspace and eigenvalue conditions hold with high probability for quite general classes of Gaussian matrices for which the predictors may be highly dependent, and hence restricted isometry conditions can be violated with high probability. In this way, our results extend the attractive theoretical guarantees on l1-relaxations to a much broader class of problems than the case of completely independent or unitary designs. We consider inexact linear equations y ≈ Φx where y is a given vector in R n , Φ is a given n x m matrix, and we wish to find x 0,∈ as sparse as possible while obeying ∥y - Φx 0,∈ ∥ 2 ≤ ∈. In general, this requires combinatorial optimization and so is considered intractable. On the other hand, the l 1 -minimization problem min ∥x∥ 1 subject to ∥y - Φx∥ 2 ≤ e is convex and is considered tractable. We show that for most Φ, if the optimally sparse approximation x 0,∈ is sufficiently sparse, then the solution x 1,∈ of the l 1 -minimization problem is a good approximation to x 0,∈ . We suppose that the columns of Φ are normalized to the unit l 2 -norm, and we place uniform measure on such Φ. We study the underdetermined case where m ∼ τn and τ > 1, and prove the existence of p = p(r) > 0 and C = C(p, τ) so that for large n and for all Φ's except a negligible fraction, the following approximate sparse solution property of Φ holds: for every y having an approximation ∥y - Φx 0 ∥ 2 ≤ ∈ by a coefficient vector x 0 e R m with fewer than ρ · n nonzeros, ∥x 1,∈ - x 0 ∥ 2 ≤ C ≤ ∈. This has two implications. First, for most Φ, whenever the combinatorial optimization result x 0,∈ would be very sparse, x 1,∈ is a good approximation to x 0,∈ . Second, suppose we are given noisy data obeying y = Φx 0 + z where the unknown x 0 is known to be sparse and the noise ∥z∥ 2 ≤ ∈. For most Φ, noise-tolerant l 1 -minimization will stably recover x 0 from y in the presence of noise z. We also study the barely determined case m = n and reach parallel conclusions by slightly different arguments. Proof techniques include the use of almost-spherical sections in Banach space theory and concentration of measure for eigenvalues of random matrices.