In [1]} the distributed subgradient descent algorithm is extended to stochastic gradients as the distributed stochastic gradient descent (DSGD) algorithm, and uses training of a CIFAR-10 classification model as a benchmark problem. One approach to improving convergence rates in distributed optimization is to introduce an auxiliary variable that estimates the global gradient. Several works make use of this mechanism, and extend it to the domain of nonconvex optimization with stochastic gradients [2]}, [3]}. The Choco-SGD algorithm for distributed deep learning [4]} is another algorithm similar to DSGD with the variations that it uses a gossip mechanism for improved consensus, and incorporates a quantization step for reducing communication bandwidth. The authors of [5]} propose using local convex approximations, based on global gradient estimates found through gradient tracking, for distributed neural network optimization and show basic results on small networks.
