GRADIENT DESCENT HAPPENS IN A TINYSUBSPACE
Guy Gur-Ari
School of Natural Sciences
Institute for Advanced Study
Princeton, NJ 08
Gradient Descent Happens in a Tiny Subspace
We show that in a variety of large-scale deep learning scenarios the gradient
dynamically converges to a very small subspace after a short period of
training. The subspace is spanned by a few top eigenvectors of the Hessian
(equal to the number of classes in the dataset), and is mostly preserved over
long periods of training. A simple argument then suggests that gradient descent
may happen mostly in this subspace. We give an example of this effect in a
solvable model of classification, and we comment on possible implications for
optimization and learning.
Guy Gur-Ari Daniel A. Roberts Ethan Dyer