\(\diamond \)  Our approach: Local SFGD.
There are two main approaches to resolve the aforementioned communication overhead.
One is to reduce the number of transmitted bits via gradient compression schemes, such as quantization [1]}, [2]} or sparsification [3]}.
The other is to increase the amount of local iterations performed on each machine, in order to reduce the total communication rounds. The latter approach is called Local SGD, and was shown to outperform (parallel) SGD in some settings [4]}, [5]}, [6]}, [7]}.
