where \(Q, K, V \in \mathbb {R}^{N \times C}\)  are the query, key and value matrices computed from input tokens; \(N\)  and \(C\)  indicate the token number and the hidden dimension. In our experiments, we adopt the DeiT [1]} as the backbone encoder, which is a data-efficient ViT variant that distills tokens to ensure the student learns from the teacher through attention.
