where \(\Vert  \cdot \Vert _2\)  denotes \(\ell _2\)  normalization, and \(l, h\)  are layer index and head index. Previous work shows that matrices constructed in this way are regarded as the specific patterns reflecting the semantic comprehension of network [1]}. And the patches encoded from input images containing high-level understanding of parts, objects, and scenes [2]}. Thus such semantic-level distillation target provide guided and meticulous supervision to the quantized ViT. The corresponding \(\tilde{G}^{l}_{{\bf q}_{h};T}\)  and \(\tilde{G}^{l}_{{\bf k}_{h};T}\)  are constructed in the same way by the teacher’s activation. Thus combining the original distillation loss in Eq. (REF ), the final distillation loss is then formulated as
\(\begin{aligned}&\mathcal {L}_{\operatorname{DGD}} = \sum _{l\in [1, L]} \sum _{h \in [1, H]} \Vert \tilde{G}^{l}_{{\bf q}_{h};T} - \tilde{G}^{l}_{{\bf q}_{h}} \Vert _2 + \Vert \tilde{G}^{l}_{{\bf k}_{h};T} - \tilde{G}^{l}_{{\bf k}_{h}} \Vert _2, \\&\mathcal {L}_{\operatorname{distillation}} = \mathcal {L}_{\operatorname{dist}} + \mathcal {L}_{\operatorname{DGD}},\end{aligned}\) 
