The representative Vision Transformer architecture [1]} first splits the input image into non-overlapped patches and projects the patches to a higher hidden dimension using one convolutional layer. The projected patches (a.k.a. tokens) are then passed through several consecutive transformer layers. Each layer contains a self-attention module and a feed-forward network (MLPs). The self-attention is computed using the scaled-dot product:
