It has been observed in a variety of contexts that gradient descent methods have great success in solving low-rank matrix factorization problems, despite the relevant problem formulation being non-convex. We tackle a particular instance of this scenario, where we seek the @math -dimensional subspace spanned by a streaming data matrix. We apply the natural first order incremental gradient descent method, constraining the gradient method to the Grassmannian. In this paper, we propose an adaptive step size scheme that is greedy for the noiseless case, that maximizes the improvement of our metric of convergence at each data index @math , and yields an expected improvement for the noisy case. We show that, with noise-free data, this method converges from any random initialization to the global minimum of the problem. For noisy data, we provide the expected convergence rate of the proposed algorithm per iteration.
This problem is non-convex firstly because of the product of the two optimization variables @math and @math and secondly because the optimization is over the Grassmannian @math , the non-convex set of all @math -dimensional subspaces in @math . However, several methods For example, the power method can solve this problem if the top @math singular values of @math are distinct @cite_24 . Specifically, considering @math , if the desired accuracy of the @math output by the power method to the global minimizer is @math , and the first two singular values of @math , @math and @math are distinct with the @math for @math , then the power method converges in @math iterations. can find the global minimizer of this problem in polynomial time under a variety of assumptions on @math .
@cite_24
1504714361
We describe two algorithms for the eigenvalue, eigenvector problem which, on input a Gaussian matrix with complex entries, finish with probability 1 and in average polynomial time.