It has been observed in a variety of contexts that gradient descent methods have great success in solving low-rank matrix factorization problems, despite the relevant problem formulation being non-convex. We tackle a particular instance of this scenario, where we seek the @math -dimensional subspace spanned by a streaming data matrix. We apply the natural first order incremental gradient descent method, constraining the gradient method to the Grassmannian. In this paper, we propose an adaptive step size scheme that is greedy for the noiseless case, that maximizes the improvement of our metric of convergence at each data index @math , and yields an expected improvement for the noisy case. We show that, with noise-free data, this method converges from any random initialization to the global minimum of the problem. For noisy data, we provide the expected convergence rate of the proposed algorithm per iteration.
In this paper, we are interested in approximating a streaming data matrix. At each step, we sample a column of @math , denoted @math . We consider the planted problem, where @math where @math is noise and @math is drawn from a continuous distribution with support on the true subspace, spanned by @math with orthonormal columns; @math , @math . When @math , we wish to find the @math that minimizes the span of the data vectors or the range of @math , denoted @math . When @math we still discuss results in terms of the distance from @math . If we consider only @math , Problem ) is identical to Problem . The GROUSE algorithm (Grassmannian Rank-One Update Subspace Estimation) we analyze is shown as Algorithm , where we generate a sequence @math of @math matrices with orthonormal columns with the goal that @math as @math . Each observed vector is used to update @math to @math , and we constrain the gradient descent method to the Grassmannian using a geodesic update @cite_21 .
@cite_21
1987142714 2132743051 2766371994 2962834831
Grassmannian rank-one update subspace estimation (GROUSE) is an iterative algorithm for identifying a linear subspace of @math Rn from data consisting of partial observations of random vectors from that subspace. This paper examines local convergence properties of GROUSE, under assumptions on the randomness of the observed vectors, the randomness of the subset of elements observed at each iteration, and incoherence of the subspace with the coordinate directions. Convergence at an expected linear rate is demonstrated under certain assumptions. The case in which the full random vector is revealed at each iteration allows for much simpler analysis and is also described. GROUSE is related to incremental SVD methods and to gradient projection algorithms in optimization. We consider the problem of reconstructing a data stream from a small subset of its entries, where the data stream is assumed to lie in a low-dimensional linear subspace, possibly corrupted by noise. It is also important to track the change of underlying subspace for many applications. This problem can be viewed as a sequential low-rank matrix completion problem in which the subspace is learned in an online fashion. The proposed algorithm, called Parallel Estimation and Tracking by REcursive Least Squares (PETRELS), identifies the underlying low-dimensional subspace via a recursive procedure for each row of the subspace matrix in parallel, and then reconstructs the missing entries via least-squares estimation if required. PETRELS outperforms previous approaches by discounting observations in order to capture long-term behavior of the data stream and be able to adapt to it. Numerical examples are provided for direction-of-arrival estimation and matrix completion, comparing PETRELS with state of the art batch algorithms. We consider the problem of learning a one-hidden-layer neural network: we assume the input @math is from Gaussian distribution and the label @math , where @math is a nonnegative vector in @math with @math , @math is a full-rank weight matrix, and @math is a noise vector. We first give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously. Inspired by the formula, we design a non-convex objective function @math whose landscape is guaranteed to have the following properties: 1. All local minima of @math are also global minima. 2. All global minima of @math correspond to the ground truth parameters. 3. The value and gradient of @math can be estimated using samples. With these properties, stochastic gradient descent on @math provably converges to the global minimum and learn the ground-truth parameters. We also prove finite sample complexity result and validate the results by simulations. This work presents GROUSE (Grassmanian Rank-One Update Subspace Estimation), an efficient online algorithm for tracking subspaces from highly incomplete observations. GROUSE requires only basic linear algebraic manipulations at each iteration, and each subspace update can be performed in linear time in the dimension of the subspace. The algorithm is derived by analyzing incremental gradient descent on the Grassmannian manifold of subspaces. With a slight modification, GROUSE can also be used as an online incremental algorithm for the matrix completion problem of imputing missing entries of a low-rank matrix. GROUSE performs exceptionally well in practice both in tracking subspaces and as an online algorithm for matrix completion.