For self-attention, \(W_Q\) , \(W_K\) , and \(W_V\)  are learned linear transformations that are shared across all spatial locations, and respectively produce queries, keys, and values when used to transform \(x\) .
Spatial geometry is captured by \(r_{a-i,b-j}\) , which is a learned relative position based embedding.
The \((W_Q x_{i j})^\top W_K x_{a b}\)  component captures the content-content interaction between the query pixel and a key pixel in the window.
The \((W_Q x_{i j})^\top r_{a-i,b-j}\)  component is the content-geometry interaction that captures the relationship between the query and the relative position of the key pixel [1]}.
Note that this formulation preserves translational equivariance.
If an object translates in an image, for any pixel within the object, the content around it stays the same, generating the same \(p_{a-i, b-j}^{i j}\) , thereby producing the same output after self-attention.
To increase expressivity, multi-headed attention [2]} is used, which repeats this computation multiple times in parallel with different parameters, analogous to group convolutions [3]}, [4]}.
