This paper shows that the solutions to various 1-norm minimization problems are unique if, and only if, a common set of conditions are satisfied. This result applies broadly to the basis pursuit model, basis pursuit denoising model, Lasso model, as well as certain other 1-norm related models. This condition is previously known to be sufficient for the basis pursuit model to have a unique solution. Indeed, it is also necessary, and applies to a variety of 1-norm related models. The paper also discusses ways to recognize unique solutions and verify the uniqueness conditions numerically. The proof technique is based on linear programming strong duality and strict complementarity results.
To our knowledge, there are few conditions addressing the solution uniqueness of problems --. The following conditions in @cite_10 , @cite_19 are sufficient for @math to the unique minimizer of for @math : However, they are not necessary as demonstrated by the following example. Let A=, b=, =1 and consider solving the Lasso problem, which is a special case of problem : 1 2 |Ax-b |_2^2+ |x |_1. One gets the solution @math and @math . However, the inequality in condition ) holds with equality. In general, conditions becomes necessary in case @math happens to be a full rank square matrix. This assumption, however, does not apply to a sparse solution @math . Nevertheless, we summarize the result in the following corollary, whose proof is given at the end of Section .
@cite_19 @cite_10
1486249389 2123202508 2155981690 2077306951
Given @math noisy samples with @math dimensions, where @math , we show that the multi-step thresholding procedure based on the Lasso -- we call it the Thresholded Lasso , can accurately estimate a sparse vector @math in a linear model @math , where @math is a design matrix normalized to have column @math norm @math , and @math . We show that under the restricted eigenvalue (RE) condition (Bickel-Ritov-Tsybakov 09), it is possible to achieve the @math loss within a logarithmic factor of the ideal mean square error one would achieve with an oracle while selecting a sufficiently sparse model -- hence achieving sparse oracle inequalities ; the oracle would supply perfect information about which coordinates are non-zero and which are above the noise level. In some sense, the Thresholded Lasso recovers the choices that would have been made by the @math penalized least squares estimators, in that it selects a sufficiently sparse model without sacrificing the accuracy in estimating @math and in predicting @math . We also show for the Gauss-Dantzig selector (Cand e s-Tao 07), if @math obeys a uniform uncertainty principle and if the true parameter is sufficiently sparse, one will achieve the sparse oracle inequalities as above, while allowing at most @math irrelevant variables in the model in the worst case, where @math is the smallest integer such that for @math , @math . Our simulation results on the Thresholded Lasso match our theoretical analysis excellently. We consider the problem of learning a coefficient vector xο ∈ RN from noisy linear observation y = Axo + ∈ Rn. In many contexts (ranging from model selection to image processing), it is desirable to construct a sparse estimator x. In this case, a popular approach consists in solving an l1-penalized least-squares problem known as the LASSO or basis pursuit denoising. For sequences of matrices A of increasing dimensions, with independent Gaussian entries, we prove that the normalized risk of the LASSO converges to a limit, and we obtain an explicit expression for this limit. Our result is the first rigorous derivation of an explicit formula for the asymptotic mean square error of the LASSO for random instances. The proof technique is based on the analysis of AMP, a recently developed efficient algorithm, that is inspired from graphical model ideas. Simulations on real data matrices suggest that our results can be relevant in a broad array of practical applications. Abstract A full-rank under-determined linear system of equations Ax = b has in general infinitely many possible solutions. In recent years there is a growing interest in the sparsest solution of this equation—the one with the fewest non-zero entries, measured by ∥ x ∥ 0 . Such solutions find applications in signal and image processing, where the topic is typically referred to as “sparse representation”. Considering the columns of A as atoms of a dictionary, it is assumed that a given signal b is a linear composition of few such atoms. Recent work established that if the desired solution x is sparse enough, uniqueness of such a result is guaranteed. Also, pursuit algorithms, approximation solvers for the above problem, are guaranteed to succeed in finding this solution. Armed with these recent results, the problem can be reversed, and formed as an implied matrix factorization problem: Given a set of vectors b i , known to emerge from such sparse constructions, Ax i = b i , with sufficiently sparse representations x i , we seek the matrix A . In this paper we present both theoretical and algorithmic studies of this problem. We establish the uniqueness of the dictionary A , depending on the quantity and nature of the set b i , and the sparsity of x i . We also describe a recently developed algorithm, the K-SVD, that practically find the matrix A , in a manner similar to the K-Means algorithm. Finally, we demonstrate this algorithm on several stylized applications in image processing. We consider the problem of estimating an unknown signal @math from noisy linear observations @math . In many practical instances, @math has a certain structure that can be captured by a structure inducing convex function @math . For example, @math norm can be used to encourage a sparse solution. To estimate @math with the aid of @math , we consider the well-known LASSO method and provide sharp characterization of its performance. We assume the entries of the measurement matrix @math and the noise vector @math have zero-mean normal distributions with variances @math and @math respectively. For the LASSO estimator @math , we attempt to calculate the Normalized Square Error (NSE) defined as @math as a function of the noise level @math , the number of observations @math and the structure of the signal. We show that, the structure of the signal @math and choice of the function @math enter the error formulae through the summary parameters @math and @math , which are defined as the Gaussian squared-distances to the subdifferential cone and to the @math -scaled subdifferential, respectively. The first LASSO estimator assumes a-priori knowledge of @math and is given by @math . We prove that its worst case NSE is achieved when @math and concentrates around @math . Secondly, we consider @math , for some @math . This time the NSE formula depends on the choice of @math and is given by @math . We then establish a mapping between this and the third estimator @math . Finally, for a number of important structured signal classes, we translate our abstract formulae to closed-form upper bounds on the NSE.