We note that although we use dual averaging as an algorithmic workhorse,
AMB-DG can be implemented using other gradient-based algorithms as
well. The motivation behind using dual averaging stems from the following.
Dual averaging has been shown to asymptotically achieve optimal regret
bound when applied in distributed optimization using delayed gradients
and fixed minibatch size [1]}. Furthermore, in [2]},
we proved that when using dual averaging in a synchronous distributed
setting with variable minibatch per epoch with no gradient staleness,
we can achieve optimal regret bound asymptotically. Hence, we anticipate
that using dual averaging with delayed gradients and variable minibatch
per epoch could yield the desired convergence rate.
