Relative position representations extend attention to consider the relative distance between tokens in an input sequence [1]}. This concept was applied to the ViT-like architecture called Swin Transformers under the name relative position bias [2]}. In practice, relative position bias is implemented using a learnable \(P \times P\)  matrix in each attention layer, where \(P\)  is the maximum number of patches. This enables additional weighting between the interaction of patches during each attention layer.
