Distributed stochastic optimization has become an important tool in
solving large-scale problems such as those found in modern machine
learning applications. Traditionally, such problems have been solved
using gradient-based methods in a serial manner on a single compute
node. However, training deep neural networks using large datasets
with millions of data points in a reasonable amount of time often
requires parallelizing across multiple machines [1]}, [2]}, [3]}, [4]}.
Furthermore, in many online learning systems, such as search engines,
data (e.g., queries) arrive at rates that a single node cannot handle
or process in real time [5]}. Although
training deep networks using a single GPU is possible, in many large-scale
training and online learning problems, multiple nodes are deployed
in parallel to cooperatively execute the desired tasks.
