The increased sizes of data sets and associated model complexities have established distributed training setups as the de facto method for training models at scale. A typical setup consists of one central machine (parameter server or PS) and multiple worker machines. The PS coordinates the protocol by communicating parameters and maintaining the global copy of the model. The workers compute gradients of the loss with respect to the optimization parameters and transmit them to the PS. The PS then updates the model. This is an iterative process and is repeated until convergence. Several frameworks including MXNet [1]}, CNTK [2]} and PyTorch [3]} support this architecture.
