Remark 1 Computation and communications. GD versus SGD. To get a handle on computation, we provide results with performing one gradient descent in FedDynOneGD algorithm. Our results show that one gradient descent step, which amounts to a single-pass through the data points in a device achieves \(\epsilon \)  expected target error with the same number of rounds compared to FedDyn as presented in Theorem REF . We may wonder whether SGD would be more beneficial in the computation metric. To understand this point note that prior works, that utilize SGD on each device, the number of computations per-device is no smaller than taking one-pass through the local dataset [1]}, [2]}, [3]}. As such SGD leads to larger noise at this level of computations, and often results in requiring a larger number of communication rounds. Furthermore, unlike GD, SGD is not parallelizable. SGD method needs to update the local model and evaluate the gradient in the updated model in a sequential manner. Differently, GD methods can evaluate the gradient by parallelizing among device datapoints. For this reason, SGD does not appear to offer any advantages over GD with respect to both communication rounds or computations per round.

