Distributed optimization methods can be categorized as synchronous
or asynchronous. In synchronous methods, the master waits for all
workers to submit their results before updating the optimization parameters.
As a result, the speed of the computation in synchronous methods is
limited by the speed of the slowest workers known as “stragglers”
[1]}, [2]}, [3]}. To alleviate the
straggler effect, asynchronous methods have been proposed in which
the master does not wait for all workers. Instead, the master updates
the optimization parameters using “stale” (i.e., delayed) gradients.
