In the SASA model of [1]}, the local window \(\mathcal {N}(i, j)\)  is a \(k\times k\)  window centered around \((i, j)\) , just like a convolution.
The size of this local window \(k\)  is an important setting to leverage in self-attention.
Unlike dense convolutions, \(k\)  can grow without significantly increasing the number of parameters.
Since the projection parameters (\(W_Q\) , \(W_K\) , \(W_V\) ) are independent of \(k\) , the only parameters that increase with \(k\)  is \(r_{a-i,b-j}\) .
However, \(r_{a-i,b-j}\)  constitutes a trivial fraction of the parameters compared to the projection parameters
For a window size as large as 63, and 16 dimensions per attention head, \(r_{a-i,b-j}\)  would add only \(63*16=1008\)  parameters per layer because \(r_{a-i,b-j}\)  are shared among heads.
In contrast, if the dimensions of the attention layer were 512, \(W_Q\) , \(W_K\) , \(W_V\)  would contribute 786432 parameters. We show details in the appendix.
, so increasing \(k\)  does not not impact the number of parameters of the layer significantly.
In contrast, the number of parameters in a convolution layer scale quadratically with \(k\)  (, a \(5\times 5\)  convolution has \(\frac{25}{9}\)  times the parameters of a \(3\times 3\)  convolution).
On the other hand, the computational cost of self-attention grows quadratically with \(k\) , preventing the use of very large values for \(k\) .
