
where the queries \(q_{ij} = W_Q x_{ij}\) , keys \(k_{ab} = W_K x_{ab}\) , and values \(v_{ab} = W_V x_{ab}\)  are linear transformations of the pixels, and \(r_{a-i,b-j}\)  is a learned relative position based embedding. Following the Transformer [1]}, we also use multihead attention, where we run multiple instances of the self-attention in parallel with different parameters. However, each head shares the parameters for the relative embeddings \(r_{a-i,b-j}\) . For an attention window of size \(k\)  around each pixel, we factorize the relative embeddings along height and width following [2]}, and we allocate half the channels within a head to each of these. Keeping the dimension per head fixed at 16 as mentioned in the paper, this gives a constant \(2(k-1) * 16\)  parameters per attention layer layer for \(r_{a-i,b-j}\) . In contrast, if the channels in an attention layer are \(d\) , then each of the three linear transformations has \(d^2\)  parameters. Thus the ratio of parameters in the relative embeddings as compared with the linear projections is \(\frac{2(k-1) * 16}{3d^2}\) , which is small for typical values of \(k\)  and \(d\) .
<TABLE><TABLE>