@inproceedings{
wang2024improving,
title={Improving Generalization and Convergence by Enhancing Implicit Regularization},
author={Mingze Wang and Jinbo Wang and Haotian He and Zilin Wang and Guanhua Huang and Feiyu Xiong and Zhiyu li and Weinan E and Lei Wu},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=cjM2bhLOiC}
}
In this work, we propose an Implicit Regularization Enhancement (IRE) framework to accelerate the discovery of flat solutions in deep learning, thereby improving generalization and convergence. 
Specifically, IRE decouples the dynamics of flat and sharp directions, which boosts the sharpness reduction along flat directions while maintaining the training stability in sharp directions. We show that IRE can be practically incorporated with *generic base optimizers* without introducing significant computational overload. Experiments show that IRE consistently improves the generalization performance for image classification tasks across a variety of benchmark datasets (CIFAR-10/100, ImageNet) and models (ResNets and ViTs). 
Surprisingly, IRE also  achieves a $2\times$ *speed-up* compared to AdamW in the pre-training of Llama models (of sizes ranging from 60M to 229M) on datasets including Wikitext-103, Minipile, and Openwebtext. Moreover, we provide theoretical guarantees, showing that IRE can substantially accelerate the convergence towards flat minima in Sharpness-aware Minimization (SAM).
Improving Generalization and Convergence by Enhancing Implicit Regularization
Mingze Wang Jinbo Wang Haotian He Zilin Wang Guanhua Huang Feiyu Xiong Zhiyu li Weinan E Lei Wu