@inproceedings{
defazio2024the,
title={The Road Less Scheduled},
author={Aaron Defazio and Xingyu Alice Yang and Ahmed Khaled and Konstantin Mishchenko and Harsh Mehta and Ashok Cutkosky},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=0XeNkkENuI}
}
Existing learning rate schedules that do not require specification of the optimization stopping step $T$ are greatly out-performed by learning rate schedules that depend on $T$. We propose an approach that avoids the need for this stopping time by eschewing the use of schedules entirely, while exhibiting state-of-the-art performance compared to schedules across a wide family of problems ranging from convex problems to large-scale deep learning problems. Our Schedule-Free approach introduces no additional hyper-parameters over standard optimizers with momentum. Our method is a direct consequence of a new theory we develop that unifies scheduling and iterate averaging. An open source implementation of our method is available at https://github.com/facebookresearch/schedule_free. Schedule-Free AdamW is the core algorithm behind our winning entry to the MLCommons 2024 AlgoPerf Algorithmic Efficiency Challenge Self-Tuning track.
The Road Less Scheduled
Aaron Defazio Xingyu Alice Yang Ahmed Khaled Konstantin Mishchenko Harsh Mehta Ashok Cutkosky