Many applications require recovering a matrix of minimal rank within an affine constraint set, with matrix completion a notable special case. Because the problem is NP-hard in general, it is common to replace the matrix rank with the nuclear norm, which acts as a convenient convex surrogate. While elegant theoretical conditions elucidate when this replacement is likely to be successful, they are highly restrictive and convex algorithms fail when the ambient rank is too high or when the constraint set is poorly structured. Non-convex alternatives fare somewhat better when carefully tuned; however, convergence to locally optimal solutions remains a continuing source of failure. Against this backdrop we derive a deceptively simple and parameter-free probabilistic PCA-like algorithm that is capable, over a wide battery of empirical tests, of successful recovery even at the theoretical limit where the number of measurements equal the degrees of freedom in the unknown low-rank matrix. Somewhat surprisingly, this is possible even when the affine constraint set is highly ill-conditioned. While proving general recovery guarantees remains evasive for non-convex algorithms, Bayesian-inspired or otherwise, we nonetheless show conditions whereby the underlying cost function has a unique stationary point located at the global optimum; no existing cost function we are aware of satisfies this same property. We conclude with a simple computer vision application involving image rectification and a standard collaborative filtering benchmark.
The IRLS0 algorithm from @cite_12 represents the best-performing special case of the above, where @math is minimized using a homotopy continuation scheme merged with IRLS. Here a fixed @math is replaced with a decreasing sequence @math , the rationale being that when @math is large, the cost function is relatively smooth and devoid of local minima. As the iterations @math progress, @math is reduced, and the cost behaves more like the matrix rank function. However, because now we are more likely to be within a reasonably good basin of attraction, spurious local minima are more easily avoided. The downside of this procedure is that it requires a pre-defined heuristic for reducing @math , and this schedule may be problem specific. Moreover, there is no guarantee that a global solution will ever be found.
@cite_12
2136912397 1967023569 2063761188 2161227280
The problem of minimizing the rank of a matrix subject to affine constraints has applications in several areas including machine learning, and is known to be NP-hard. A tractable relaxation for this problem is nuclear norm (or trace norm) minimization, which is guaranteed to find the minimum rank matrix under suitable assumptions. In this paper, we propose a family of Iterative Reweighted Least Squares algorithms IRLS-p (with 0 ≤ p ≤ 1), as a computationally efficient way to improve over the performance of nuclear norm minimization. The algorithms can be viewed as (locally) minimizing certain smooth approximations to the rank function. When p = 1, we give theoretical guarantees similar to those for nuclear norm minimization, that is, recovery of low-rank matrices under certain assumptions on the operator defining the constraints. For p < 1, IRLS-p shows better empirical performance in terms of recovering low-rank matrices than nuclear norm minimization. We provide an efficient implementation for IRLS-p, and also present a related family of algorithms, sIRLS-p. These algorithms exhibit competitive run times and improved recovery when compared to existing algorithms for random instances of the matrix completion problem, as well as on the MovieLens movie recommendation data set. This paper presents a way of using the Iteratively Reweighted Least Squares (IRLS) method to minimize several robust cost functions such as the Huber function, the Cauchy function and others. It is known that IRLS (otherwise known as Weiszfeld) techniques are generally more robust to outliers than the corresponding least squares methods, but the full range of robust M-estimators that are amenable to IRLS has not been investigated. In this paper we address this question and show that IRLS methods can be used to minimize most common robust M-estimators. An exact condition is given and proved for decrease of the cost, from which convergence follows. In addition to the advantage of increased robustness, the proposed algorithm is far simpler than the standard L1 Weiszfeld algorithm. We show the applicability of the proposed algorithm to the rotation averaging, triangulation and point cloud alignment problems. In this paper we study general @math l p regularized unconstrained minimization problems. In particular, we derive lower bounds for nonzero entries of the first- and second-order stationary points and hence also of local minimizers of the @math l p minimization problems. We extend some existing iterative reweighted @math l 1 ( @math IRL 1 ) and @math l 2 ( @math IRL 2 ) minimization methods to solve these problems and propose new variants for them in which each subproblem has a closed-form solution. Also, we provide a unified convergence analysis for these methods. In addition, we propose a novel Lipschitz continuous @math ∈ -approximation to @math ? x ? p p . Using this result, we develop new @math IRL 1 methods for the @math l p minimization problems and show that any accumulation point of the sequence generated by these methods is a first-order stationary point, provided that the approximation parameter @math ∈ is below a computable threshold value. This is a remarkable result since all existing iterative reweighted minimization methods require that @math ∈ be dynamically updated and approach zero. Our computational results demonstrate that the new @math IRL 1 method and the new variants generally outperform the existing @math IRL 1 methods (Chen and Zhou in 2012; Foucart and Lai in Appl Comput Harmon Anal 26:395---407, 2009). We consider solving the @math -regularized least-squares ( @math -LS) problem in the context of sparse recovery for applications such as compressed sensing. The standard proximal gradient method, also known as iterative soft-thresholding when applied to this problem, has low computational cost per iteration but a rather slow convergence rate. Nevertheless, when the solution is sparse, it often exhibits fast linear convergence in the final stage. We exploit the local linear convergence using a homotopy continuation strategy, i.e., we solve the @math -LS problem for a sequence of decreasing values of the regularization parameter, and use an approximate solution at the end of each stage to warm start the next stage. Although similar strategies have been studied in the literature, there have been no theoretical analysis of their global iteration complexity. This paper shows that under suitable assumptions for sparse recovery, the proposed homotopy strategy ensures that all iterates along the homotopy sol...