As an optimization technique based on element-level comparison of activation, distillation allows the quantized ViT to mimic the full-precision teacher model about output logits. However, we find that the distillation procedure used in previous ViT and fully quantized ViT baseline (Sec. ) unable to deliver meticulous supervision to attention weights (shown in Fig. REF ), leading to insufficient optimization. To solve the optimization insufficiency in the distillation of the fully quantized ViT, we propose the Distribution-Guided Distillation (DGD) method in Q-ViT.
We first build patch-based similarity pattern matrices for distilling the upstream query and key instead of attention following [1]}, which is formulated as
\(\begin{aligned}\tilde{G}^{l}_{{\bf q}_{h}} = \tilde{\bf q}^{l}_{h} \cdot ({\tilde{\bf q}^{l}_{h}})^{\top }, \;\; G^{(l)}_{{\bf q}_{h}} = \tilde{G}^{l}_{{\bf q}_{h}} / \Vert  \tilde{G}^{l}_{{\bf q}_{h}}\Vert _2, \\\tilde{G}^{l}_{{\bf k}_{h}} = \tilde{\bf k}^{l}_{h} \cdot ({\tilde{\bf k}^{l}_{h}})^{\top }, \;\; G^{(l)}_{{\bf k}_{h}} = \tilde{G}^{l}_{{\bf k}_{h}} / \Vert  \tilde{G}^{l}_{{\bf k}_{h}}\Vert _2,\end{aligned}\) 
