Before entering the ViT, each patch encodes its position in the original image stored via position embedding. In theory, this allows for permutation of patches without impacting the performance of the model. In addition to position embeddings, Transformers and ViTs may use a relative position bias [1]}, [2]}, which learns to encode the relative distance between each pair of patches. For patch sampling techniques, such as our proposed PSS, we find that the inclusion of the relative position bias leads to dramatic improvement in performance.
