From (REF ), one can see that SFGD is amenable to parallelization, simply by replacing \(H(U_t)\)  with an average of stochastic gradients that are computed independently from the local machines.
This scheme is often the state-of-the-art in distributed learning problems [1]}, [2]}.
However, it exhibits a major drawback: on every iteration, each machine has to send the local (stochastic) gradient to the server, and receive back the aggregated model parameter. Typically, such communication is much more expensive than the local computations that each machine has to perform [3]}.
