@inproceedings{
tahmasebi2023on,
title={On Scale-Invariant Sharpness Measures},
author={Behrooz Tahmasebi and Ashkan Soleymani and Stefanie Jegelka and Patrick Jaillet},
booktitle={NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning},
year={2023},
url={https://openreview.net/forum?id=Shqnglu4En}
}
Recently, there has been a substantial surge of interest in the development of optimization algorithms tailored for overparameterized models. This interest centers around the objective of minimizing a concept of sharpness in conjunction with the original loss function, e.g., the Sharpness-Aware Minimization (SAM) algorithm shown effective in practice. Nevertheless, the majority of sharpness measures exhibit sensitivity to parameter scaling in neural networks, and they may even experience significant magnification when subjected to rescaling operations. Motivated by this issue, in this paper, we introduce a new class of scale-invariant sharpness measures, that gives rise to a new class of scale-invariant sharpness-aware objective functions.  Furthermore, we prove that the newly introduced objective functions are explicitly biased towards the minimization of our scale-invariant sharpness measures.
On Scale-Invariant Sharpness Measures
Behrooz Tahmasebi Ashkan Soleymani Stefanie Jegelka Patrick Jaillet