Optimization bottleneck.  We calculate l2-norm distances between each attention weight among different blocks in DeiT-S architecture as shown in Fig. REF . The MHSA modules in full-precision ViT with different depth learn different representations from images. As mentioned in [1]}, lower ViT layers attend both locally and globally while higher ViT layers pay most attention to global representations. However, the fully quantized ViT (blue lines in Fig. REF ) fails to learn accurate attention map distances. Thus, it requires a new design that could utilize the full-precision teacher’s information better.
