Substantial efforts have been made to compress and accelerate neural networks for efficient online inference. Methods include compact network design [1]}, network pruning [2]}, low-rank decomposition [3]}, quantization [4]}, [5]}, [6]}, and knowledge distillation [7]}, [8]}. Quantization is particularly suitable for deployment on AI chips because it reduces the bit-width of network parameters and activations for efficient inference. Prior post-training quantization (PTQ) methods [9]}, [10]} on ViTs directly compute quantized parameters based on pre-trained full-precision models, which constrains the model performance to a sub-optimized level without fine-tuning. Furthermore, quantizing these models based on PTQ methods to ultra-low bits (e.g., 4 bits or lower) is ineffective and suffers from a significant performance reduction.
<FIGURE>