@inproceedings{
singh2023moxcohow,
title={Mo{XC}o:How I learned to stop exploring and love my local minima?},
author={Esha Singh and Shoham Sabach and Yu-Xiang Wang},
booktitle={NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning},
year={2023},
url={https://openreview.net/forum?id=8s8w7nwCuk}
}
Deep Neural Networks (DNNs) are well-known for their generalization capabilities despite overparameterization. This is commonly attributed to the optimizer’s ability to find “good” solutions within high-dimensional loss landscapes. However, widely employed adaptive optimizers, such as ADAM, may suffer from subpar generalization. This paper presents an innovative methodology, $\textit{MoXCo}$, to address these concerns by designing adaptive optimizers that not only expedite exploration with faster convergence speeds but also ensure the avoidance of over-exploitation in specific parameter regimes, ultimately leading to convergence to good solutions.
MoXCo:How I learned to stop exploring and love my local minima?
Esha Singh Shoham Sabach Yu-Xiang Wang