Poseidon An Efficient Communication Interface for Distributed Deep Learning on GPU Clusters
Deep learning models can take weeks to train on a single GPU equipped machine, necessitating scaling out DL training to a GPU cluster. However, current distributed DL implementations can scale poorly due to substantial parameter synchronization over the network, because the high throughput of GPUs allows more data batches to be processed per unit time than CPUs, leading to more frequent network synchronization. We present Poseidon, an efficient communication interface for distributed DL on GPUs. Poseidon exploits the layered model structures in DL programs to overlap communication and computation, reducing bursty network communication. Moreover, Poseidon uses a hybrid communication scheme that optimizes the number of bytes required to synchronize each layer, according to layer properties and the number of machines. We show that Poseidon is applicable to different DL frameworks, by implementing Poseidon into Caffe and TensorFlow, and showing that, with Poseidon, both Caffe and TensorFlow can achieve 15.5x speed up on VGG19 22K network for image classification using 16 single GPU machines, even under limited bandwidth (10GbE) In particular, Poseidon enabled TensorFlow achieves 31.5x speed up with 32 single GPU machines on Inception V3, a 50% improvement over the original TensorFlow (20x speed up) The software is available at http:/poseidon release.readthedocs.io.