@inproceedings{
lu2023benign,
title={Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate},
author={Miao Lu and Beining Wu and Xiaodong Yang and Difan Zou},
booktitle={NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning},
year={2023},
url={https://openreview.net/forum?id=FM6MtmeRxZ}
}
In this work, we theoretically investigate the generalization properties of neural networks (NN) trained by stochastic gradient descent (SGD) with \emph{large learning rates}. Under such a training regime, our finding is that, the \emph{oscillation} of the NN weights caused by SGD with large learning rates turns out to be beneficial to the generalization of the NN, potentially improving over the same NN trained by SGD with small learning rates that converges more smoothly. In view of this finding, we call such a phenomenon ``\emph{benign oscillation}".
Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate
Miao Lu Beining Wu Xiaodong Yang Difan Zou