ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
How to efficiently serve ever-larger trained natural language models in
practice has become exceptionally challenging even for powerful cloud servers
due to their prohibitive memory/computation requirements. In this work, we
present an efficient and affordable post-training quantization approach to
compress large Transformer-based models, termed as ZeroQuant. ZeroQuant is an
end-to-end quantization and inference pipeline with three main components: (1)
a fine-grained hardware-friendly quantization scheme for both weight and
activations; (2) a novel affordable layer-by-layer knowledge distillation
algorithm (LKD) even without the access to the original training data; (3) a
highly-optimized quantization system backend support to remove the
quantization/dequantization overhead. As such, we are able to show that: (1)
ZeroQuant can reduce the precision for weights and activations to INT8 in a
cost-free way for both BERT and GPT3-style models with minimal accuracy impact,
which leads to up to 5.19x/4.16x speedup on those models compared to FP16
inference; (2) ZeroQuant plus LKD affordably quantize the weights in the
fully-connected module to INT4 along with INT8 weights in the attention module
and INT8 activations, resulting in 3x memory footprint reduction compared to
the FP16 model; (3) ZeroQuant can be directly applied to two of the largest
open-sourced language models, including GPT-J6B and GPT-NeoX20, for which our
INT8 model achieves similar accuracy as the FP16 model but achieves up to 5.2x
better efficiency.