In specific, for the design of VTM, we extend the original 2D version of swin transformer [1]}.
As illustrated in Fig. REF , we first evenly partition query and support spatial dimensions of \(\mathcal {M}_p\)  into non-overlapping sub-hypercorrelations \(\mathcal {M}^{\prime }_p \in \mathbb {R}^{n\times n \times n \times n \times D}\) . We compute self-attention within each partitioned sub-hypercorrelation. Subsequently, we shift the windows by displacement of \((\lfloor \frac{n}{2} \rfloor ,\lfloor \frac{n}{2} \rfloor ,\lfloor \frac{n}{2} \rfloor ,\lfloor \frac{n}{2} \rfloor )\)  pixels from the previously partitioned windows, which we perform self-attention within the newly created windows. Then as done in original swin transformer [1]}, we simply roll it back to its original form without adopting any complex implementation. In computing self-attention, we use relative position bias and take the values from an expanded parameterized bias matrix, following [3]}, [4]}, [1]}. We leave other components in swin transformer blocks unchanged, e.g., Layer Normalization (LN) [6]} and MLP layers.
