@inproceedings{
yaras2023invariant,
title={Invariant Low-Dimensional Subspaces in Gradient Descent for Learning Deep Matrix Factorizations},
author={Can Yaras and Peng Wang and Wei Hu and Zhihui Zhu and Laura Balzano and Qing Qu},
booktitle={NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning},
year={2023},
url={https://openreview.net/forum?id=4pPnQqUMLS}
}
An extensively studied phenomenon of the past few years in training deep networks is the implicit bias of gradient descent towards parsimonious solutions. In this work, we further investigate this phenomenon by narrowing our focus to deep matrix factorization, where we reveal surprising low-dimensional structures in the learning dynamics when the target matrix is low-rank. Specifically, we show that the evolution of gradient descent starting from arbitrary orthogonal initialization only affects a minimal portion of singular vector spaces across all weight matrices. In other words, the learning process happens only within a small invariant subspace of each weight matrix, despite the fact that all parameters are updated throughout training. From this, we provide rigorous justification for low-rank training in a specific, yet practical setting. In particular, we demonstrate that we can construct compressed factorizations that are equivalent to full-width, deep factorizations throughout training for solving low-rank matrix completion problems efficiently.
Invariant Low-Dimensional Subspaces in Gradient Descent for Learning Deep Matrix Factorizations
Can Yaras Peng Wang Wei Hu Zhihui Zhu Laura Balzano Qing Qu