Communication Efficient Distributed Deep Learning with GPU FPGA Heterogeneous Computing
Distributed deep learning has an inevitable bottleneck in communication between GPUs. This bottleneck is due to the difference in the data transfer latency of internode and intra node communication. Here, we propose a GPU FPGA (field programmable gate array) heterogeneous computing system and a hardware specific communication module with parameter based computation/communication overlap. In our system, we configure a network interface card (NIC) with a data aggregation function. To reduce the load on CPUs, we developed a device driver for remote direct memory access between GPUs and FPGAs. The hardware and software can be run with TensorFlow and Horovod. For comparison, we compared our system with the conventional GPUDirect RDMA system. Results of the measurement show Allreduce latency is reduced to 1/4 by offloading Allreduce to FPGA NICs. Our hardware/software co design can also conceal about 90% of the communication overhead and improve scalability by 20% The end to end time consumed for training in distributed deep learning with ResNet 50 and ImageNet is reduced to 87.3% without any degradation in validation accuracy.
2020 IEEE Symposium on High Performance Interconnects (HOTI)