In a similar vein, there are works that augment updates with extra device variables that are also transmitted along with the models [1]}, [2]}. These works prove convergence guarantees through adding device-dependent regularizers. Nevertheless, they suffer additional communication costs and they are not extensively experimented with deep neural networks. Among them, SCAFFOLD [1]} is a closely related work even though it transmits extra variables and a more detailed comparison is given in Section .
