In the past decade, a large body of work has proposed and analyzed
various schemes to parallelize stochastic optimization and online
learning problems [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}, [12]}, [13]}, [14]}, [15]}.
A common approach is to use a network of nodes arranged according
to a hub-and-spoke topology. In this topology, a central node,
known as the master, keeps track of and updates the optimization
parameters while the other nodes, known as workers, calculate
the gradients of the objective function with respect to individual
data points and submit these gradients to the master.
