Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression
Recent advances in distributed optimization and learning have shown that
communication compression is one of the most effective means of reducing
communication. While there have been many results on convergence rates under
communication compression, a theoretical lower bound is still missing.
  Analyses of algorithms with communication compression have attributed
convergence to two abstract properties: the unbiased property or the
contractive property. They can be applied with either unidirectional
compression (only messages from workers to server are compressed) or
bidirectional compression. In this paper, we consider distributed stochastic
algorithms for minimizing smooth and non-convex objective functions under
communication compression. We establish a convergence lower bound for
algorithms whether using unbiased or contractive compressors in unidirection or
bidirection. To close the gap between the lower bound and the existing upper
bounds, we further propose an algorithm, NEOLITHIC, which almost reaches our
lower bound (up to logarithm factors) under mild conditions. Our results also
show that using contractive bidirectional compression can yield iterative
methods that converge as fast as those using unbiased unidirectional
compression. The experimental results validate our findings.