The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon
The grokking phenomenon as reported by Power et al. ( arXiv:2201.02177 )
refers to a regime where a long period of overfitting is followed by a
seemingly sudden transition to perfect generalization. In this paper, we
attempt to reveal the underpinnings of Grokking via a series of empirical
studies. Specifically, we uncover an optimization anomaly plaguing adaptive
optimizers at extremely late stages of training, referred to as the Slingshot
Mechanism. A prominent artifact of the Slingshot Mechanism can be measured by
the cyclic phase transitions between stable and unstable training regimes, and
can be easily monitored by the cyclic behavior of the norm of the last layers
weights. We empirically observe that without explicit regularization, Grokking
as reported in ( arXiv:2201.02177 ) almost exclusively happens at the onset of
Slingshots, and is absent without it. While common and easily reproduced in
more general settings, the Slingshot Mechanism does not follow from any known
optimization theories that we are aware of, and can be easily overlooked
without an in depth examination. Our work points to a surprising and useful
inductive bias of adaptive gradient optimizers at late stages of training,
calling for a revised theoretical analysis of their origin.