Although VEM reduces the number of tokens to some extent, directly applying standard transformers [1]} that has quadratic complexity with respect to number of tokens is still challenging. Our Volumetric Transformer Module (VTM) tackles this by extending swin transformer [2]}. By setting a local 4D window, the computation for computing self-attention can be significantly reduced, without significant drop in performance. While [3]}, [4]}, [5]} could also handle a long sequence of tokens with linear complexity, we argue that as proven in optical flow and semantic correspondence literature [6]}, [7]} that neighboring pixels tend to have similar correspondences, computing self-attention over local regions and allowing cross-window interaction can help to find reliable correspondences, which in turn yields better segmentation performance in our framework. It should be noted that use of 4D convolutions [8]} also considers local consensus, but it lacks an ability to consider pixel-wise interactions due to the use of fixed kernels during convolution while transformers attentively explore pixel-wise interactions, which we validate our choice in later section.
