Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees
Communication compression is a crucial technique for modern distributed
learning systems to alleviate their communication bottlenecks over slower
networks. Despite recent intensive studies of gradient compression for data
parallel-style training, compressing the activations for models trained with
pipeline parallelism is still an open problem. In this paper, we propose
AC-SGD, a novel activation compression algorithm for communication-efficient
pipeline parallelism training over slow networks. Different from previous
efforts in activation compression, instead of compressing activation values
directly, AC-SGD compresses the changes of the activations. This allows us to
show, to the best of our knowledge for the first time, that one can still
achieve $O(1/\sqrt{T})$ convergence rate for non-convex objectives under
activation compression, without making assumptions on gradient unbiasedness
that do not hold for deep learning models with non-linear activation
functions.We then show that AC-SGD can be optimized and implemented
efficiently, without additional end-to-end runtime overhead.We evaluated AC-SGD
to fine-tune language models with up to 1.5 billion parameters, compressing
activations to 2-4 bits.AC-SGD provides up to 4.3X end-to-end speed-up in
slower networks, without sacrificing model quality. Moreover, we also show that
AC-SGD can be combined with state-of-the-art gradient compression algorithms to
enable "end-to-end communication compression: All communications between
machines, including model gradients, forward activations, and backward
gradients are compressed into lower precision.This provides up to 4.9X
end-to-end speed-up, without sacrificing model quality.