We study distributed optimization algorithms for minimizing the average of convex functions. The applications include empirical risk minimization problems in statistical machine learning where the datasets are large and have to be stored on different machines. We design a distributed stochastic variance reduced gradient algorithm that, under certain conditions on the condition number, simultaneously achieves the optimal parallel runtime, amount of communication and rounds of communication among all distributed first-order methods up to constant factors. Our method and its accelerated extension also outperform existing distributed algorithms in terms of the rounds of communication as long as the condition number is not too large compared to the size of data in each machine. We also prove a lower bound for the number of rounds of communication for a broad class of distributed first-order methods including the proposed algorithms in this paper. We show that our accelerated distributed stochastic variance reduced gradient algorithm achieves this lower bound so that it uses the fewest rounds of communication among all distributed first-order algorithms.
Building on the work of @cite_19 , we prove a new lower bound showing that any distributed first-order algorithm requires @math rounds. This lower bound combined with the convergence analysis of DASVRG shows that DASVRG is optimal in the number of rounds of communication.
@cite_19
1518479949
We introduce a novel lower bound technique for distributed graph algorithms under bandwidth limitations. We define the notion of and exemplify its strength by proving two new lower bounds for triangle membership in the CONGEST(B) model: (i) Any @math -round algorithm requires @math for a constant @math . (ii) If @math , even in constant-degree graphs any algorithm must take @math rounds. The implication of the former is the first proven separation between the LOCAL and the CONGEST models for deterministic triangle membership. The latter result is the first non-trivial lower bound on the number of rounds required, even for , under limited bandwidth. All previous known techniques are provably incapable of giving these bounds. We hope that our approach may pave the way for proving lower bounds for additional problems in various settings of distributed computing for which previous techniques do not suffice.