In this section we demonstrate the performance of AMB-DG and compare
it to other schemes. We evaluate the performance in two ways. First,
we solve a linear regression problem using a synthetic dataset. We
model the per-node compute time using the shifted exponential distribution.
We compare the performance of AMB-DG with that of AMB under the assumption
of long communication delays. Furthermore, we compare the performance
of AMB-DG with a fixed minibatch delayed-gradient-based scheme called
\(K\) -batch async [1]}, [2]}.
\(K\) -batch async is a suitable baseline to compare AMB-DG with since
both schemes ensure that workers never idle. Hence, they both use
stale gradients. Moreover, both schemes exploit all jobs completed
by all workers. In \(K\) -batch async, in each epoch each worker calculates
\(\tilde{b}/K\)  gradients and submits their sum to the master. Here
\(\tilde{b}\)  is the per-epoch fixed minibatch size and \(K\)  is an
integer that divides \(\tilde{b}\) . The master updates the parameter
\(w\)  after receiving \(K\)  messages from the workers. These \(K\)  messages
need not be from distinct workers. Faster workers can contribute more
messages. Second, we compare the performance of AMB-DG with that of
\(K\) -batch async when training a neural network to classify images
from the CIFAR-10 dataset [3]}. We run these
neural network experiments on SciNet which is an academic high performance
computing cluster. All experiments are repeated 10 times and the
average performance is reported. We note that the purpose of these
experiments is to compare the performance of our approach that is
based on fixing the compute time with the that of the state-of-the-art
approach that uses a fixed minibatch per epoch. Methods that speed
up convergence when using delayed gradients such as those found in
[4]} and [5]}
can also be applied to AMB-DG.
