Aggregating the hypercorrelation by considering all the query and support spatial dimensions, i.e., \(h_{q} \times w_{q} \times h_{s} \times w_{s}\) , as tokens requires extremely large computation, which has to be resolved to fully exploit all the pixel-wise interactions present in the correlation maps. Perhaps the most straightforward way to reduce the resolutions is to use 4D spatial pooling across query and support spatial dimensions, but this strategy risks losing some information. As an alternative, one can split the hypercorrelation into non-overlapping tensors and embed with a large learnable kernel similarly to a patch embedding in ViT [1]}, but this demands a substantial amount of resources due to the curse of dimensionality. Furthermore, unlike image classification task [1]} which finds a global representation of an image, segmentation aiming for dense prediction needs to consider overlapping neighborhood information.
