@inproceedings{
franke2023new,
title={New Horizons in Parameter Regularization: A Constraint Approach},
author={J{\"o}rg K.H. Franke and Michael Hefenbrock and Gregor Koehler and Frank Hutter},
booktitle={OPT 2023: Optimization for Machine Learning},
year={2023},
url={https://openreview.net/forum?id=AyzhPfICBX}
}
This work presents constrained parameter regularization (CPR), an alternative to traditional weight decay. Instead of applying a constant penalty uniformly to all parameters, we enforce an upper bound on a statistical measure (e.g., the L2-norm) of individual parameter groups. This reformulates learning as a constrained optimization problem. To solve this, we utilize an adaptation of the augmented Lagrangian method. Our approach allows for varying regularization strengths across different parameter groups, removing the need for explicit penalty coefficients in the regularization terms. CPR only requires two hyperparameters and introduces no measurable runtime overhead. We offer empirical evidence of CPR's effectiveness through experiments in the "grokking" phenomenon, object detection, and language modeling. Our findings show that CPR can counteract the effects of grokking, and it consistently matches or surpasses the performance of traditional weight decay.
New Horizons in Parameter Regularization: A Constraint Approach
Jörg K.H. Franke Michael Hefenbrock Gregor Koehler Frank Hutter