Communication-Efficient Distributionally Robust Decentralized Learning
Decentralized learning algorithms empower interconnected edge devices to
share data and computational resources to collaboratively train a machine
learning model without the aid of a central coordinator (e.g. an orchestrating
basestation). In the case of heterogeneous data distributions at the network
devices, collaboration can yield predictors with unsatisfactory performance for
a subset of the devices. For this reason, in this work we consider the
formulation of a distributionally robust decentralized learning task and we
propose a decentralized single loop gradient descent/ascent algorithm (AD-GDA)
to solve the underlying minimax optimization problem. We render our algorithm
communication efficient by employing a compressed consensus scheme and we
provide convergence guarantees for smooth convex and non-convex loss functions.
Finally, we corroborate the theoretical findings with empirical evidence of the
ability of the proposed algorithm in providing unbiased predictors over a
network of collaborating devices with highly heterogeneous data distributions.