@inproceedings{
d'angelo2023why,
title={Why Do We Need Weight Decay for Overparameterized Deep Networks?},
author={Francesco D'Angelo and Aditya Varre and Maksym Andriushchenko and Nicolas Flammarion},
booktitle={NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning},
year={2023},
url={https://openreview.net/forum?id=3vBtKsiHEC}
}
Weight decay is a broadly used technique for training state-of-the-art deep networks. Despite its widespread usage, its role remains poorly understood. In this work, we highlight that the role of weight decay in modern deep learning is different from its regularization effect studied in classical learning theory. For overparameterized deep networks, we show how weight decay modifies the optimization dynamics enhancing the ever-present implicit regularization of SGD via loss stabilization.
Why Do We Need Weight Decay for Overparameterized Deep Networks?
Francesco D'Angelo Aditya Varre Maksym Andriushchenko Nicolas Flammarion