We consider whether algorithmic choices in over-parameterized linear matrix factorization introduce implicit regularization. We focus on noiseless matrix sensing over rank- @math positive semi-definite (PSD) matrices in @math , with a sensing mechanism that satisfies the restricted isometry property (RIP). The algorithm we study is that of , where we model the low-rankness and PSD constraints with the factorization @math , where @math . Surprisingly, recent work argues that the choice of @math is not pivotal: even setting @math is sufficient for factored gradient descent to find the rank- @math solution, which suggests that operating over the factors leads to an implicit regularization. In this note, we provide a different perspective. We show that, in the noiseless case, under certain conditions, the PSD constraint by itself is sufficient to lead to a unique rank- @math matrix recovery, without implicit or explicit low-rank regularization. , under assumptions, the set of PSD matrices, that are consistent with the observed data, is a singleton, irrespective of the algorithm used.
We study the problem of recovering a low-rank matrix @math from linear measurements using an over-parameterized model. We parameterize the rank- @math matrix @math by @math where @math is a square matrix, whereas the number of linear measurements is much less than @math . We show that with @math random linear measurements, the gradient descent on the squared loss, starting from a small initialization, recovers @math approximately in @math iterations. The results solve the conjecture of under the restricted isometry property, and demonstrate that the training algorithm can provide an implicit regularization for non-linear matrix factorization models. We study implicit regularization when optimizing an underdetermined quadratic objective over a matrix @math with gradient descent on a factorization of X. We conjecture and provide empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent on a full dimensional factorization converges to the minimum nuclear norm solution. We provide new theoretical insights on why over-parametrization is effective in learning neural networks. For a @math hidden node shallow network with quadratic activation and @math training data points, we show as long as @math , over-parametrization enables local search algorithms to find a optimal solution for general smooth and convex loss functions. Further, despite that the number of parameters may exceed the sample size, using theory of Rademacher complexity, we show with weight decay, the solution also generalizes well if the data is sampled from a regular distribution such as Gaussian. To prove when @math , the loss function has benign landscape properties, we adopt an idea from smoothed analysis, which may have other applications in studying loss surfaces of neural networks.