where \(f(i, j, a, b)\)  is a function that returns a weight matrix \(W \in \mathcal {R}^{c_{in} \times c_{out}}\)  at every location in a 2D window \(\mathcal {N}(i, j)\)  of size \(k \times k\)  centered at \((i, j)\) .
Note that later in this section, we introduce non-centered windows for self-attention, but we use centering here for ease of explanation.
This computation is repeated for every pixel \((i, j)\) .
For a convolution, \(f(i, j, a, b)\)  returns a different linear transformation for each relative distance in neighborhood, and these weights are shared across all \((i, j)\) .
Weight sharing significantly reduces parameters and encourages learning features that repeat spatially.
In dot-product relative self-attention [1]}, [2]}, [3]} (eq:self-att-func-1,eq:self-att-func-2), every pixel in the neighborhood shares the same linear transformation which is multiplied by a scalar probability that is a function of both content-content and content-geometry interactions resulting in weights that can vary spatially.
As an example, for a ball and an orange at two different locations in an image, pixels inside the ball and the orange are likely to generate different \(p_{a-i, b-j}^{i j}\)  because of the different content around them, such as color or texture.
\(f(i, j, a, b)^{conv} & = W_{a-i, b-j}  \\\begin{split}f(i, j, a, b)^{self-att} & = \texttt {softmax}_{a b}\Big ((W_Q x_{i j})^\top W_K x_{a b} + \\& \quad (W_Q x_{i j})^\top r_{a-i,b-j} \Big ) W_V \end{split} \\& = p_{a-i, b-j}^{i j} W_v \) 
