How to efficiently serve ever-larger trained natural language models in practice has become exceptionally challenging even for powerful cloud servers due to their prohibitive memory/computation requirements.
In this work, we present an efficient and affordable post-training quantization approach to compress large Transformer-based models, termed as ZeroQuant.
ZeroQuant is an end-to-end quantization and inference pipeline with three main components:
(1)  a fine-grained hardware-friendly quantization scheme for both weight and activations;
(2)  a novel affordable layer-by-layer knowledge distillation algorithm (LKD)  even without the access to the original training data;
(3)  a highly-optimized quantization system backend support to remove the quantization/dequantization overhead.
As such, we are able to show that:
(1)  ZeroQuant can reduce the precision for weights and activations to INT8 in a cost-free way for both BERT and GPT-3-style models with minimal accuracy impact, which leads to up to 5.19x/4.16x speedup on those models compared to FP16 inference;
(2)  ZeroQuant plus LKD affordably quantize the weights in the fully-connected module to INT4 along with INT8 weights in the attention module and INT8 activations, resulting in 3x memory footprint reduction compared to the FP16 model;
(3)  ZeroQuant can be directly applied to two of the largest open-sourced language models, including GPT-Jand GPT-NeoX, for which our INT8 model achieves similar accuracy as the FP16 model but achieves up to 5.2x better efficiency.