@inproceedings{
dahan2024slowcalsgd,
title={{SL}owcal{SGD} : Slow Query Points Improve Local-{SGD} for Stochastic Convex Optimization},
author={Tehila Dahan and Kfir Yehuda Levy},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=B29BlRe26Z}
}
We consider  distributed learning scenarios where $M$ machines interact with a parameter server along several communication rounds in order to minimize a joint objective function. 
Focusing on the heterogeneous case, where different machines may draw samples from different data-distributions, we design the first local update method that provably benefits over the two most prominent distributed baselines: namely Minibatch-SGD and Local-SGD. 
Key to our approach is a slow querying technique  that we customize to the distributed setting, which in turn enables a better mitigation of the bias caused by local updates.
SLowcalSGD : Slow Query Points Improve Local-SGD for Stochastic Convex Optimization
Tehila Dahan Kfir Yehuda Levy