Differently, quantization-aware training (QAT) [1]} methods perform quantization during back propagation and achieve much less performance drop with a higher compression rate generally. QAT is shown to be effective for CNN models [2]} for CV tasks. However, QAT methods remain largely unexplored for low-bit quantization of vision transformers. Therefore, we first build a fully quantized ViT baseline, a straightforward yet effective solution based on common techniques. Our study discovers that the performance drop of fully quantized ViT lies in the information distortion among the attention mechanism in the forward process, and the ineffective optimization for eliminating the distribution difference through distillation in the backward propagation.
First, the attention mechanism of ViT aims at modeling long-distance dependencies [3]}, [4]}.
However, our analysis shows that a direct
quantization method leads to the information distortion, i.e., significant distribution variation for the query module between quantized ViT and full-precision counterpart. For example, as shown in Fig. REF , the variance difference is 0.4409 (1.2124 v.s. 1.6533) for the first block The Gaussian distribution hypothesis is supported by [5]}. This inevitably deteriorates the representation capability of the attention module on capturing the global dependency for the input.
Second, the distillation for the fully quantized ViT baseline utilizes distillation token (following [6]}) to directly supervise the classification output of the quantized ViT.
However, we found that such a simple supervision is ineffective, which is coarse-grained for the large gap between the quantized attention scores and their full-precision counterparts.
