The canonical Vision Transformer (ViT) uses self-attention and MLPs as its backbone [1]}. In particular, a single “head" of a self-attention network is given by the following:
\(f_j(\mathbf {X}_i) := \sigma \left(\frac{\mathbf {X}_i \mathbf {Q}_j \mathbf {K}_j^\top \mathbf {X}_i^\top }{\sqrt{d}}\right) \mathbf {X}_i \mathbf {V}_j,\) 
