

Appendix
In this document, we provide detailed implementation details, more quantitative results on semantic correspondence benchmarks, including SPair-71k [1]}, PF-PASCAL [2]}, and PF-WILLOW [3]}, and more qualitative results on all the benchmarks we used.
Appendix A. Implementation details
Implementation Details.
We first extract backbone features using ResNet50 [4]} and ResNet101 [4]} from conv3\(\_\) x, conv4\(\_\) x and conv5\(\_\) x. This feature extracting scheme results in 3 pyramidal layers. To compose VEM, we stack a series of \(3\times 3 \times 3 \times 3\)  convolutions followed by ReLU and Group Normalization. The coarsest level correaltion map has a size of \(8\times 8 \times 8 \times 8\)  after undergoing VEM. Finer levels, \(p=4\)  and \(p=3\) , have volume embeddings of size \(16\times 16 \times 8 \times 8\)  and \(32 \times 32 \times 8 \times 8\) , respectively. We set the number of heads in VTM as 4. At \(p=5\)  level, we set the depth of VTM as 4 and rest are set to 2. Window size \(M\)  is set to 4 at all levels of VTM. We project the feature maps of query at last layer of level \(p=4, p=3\)  and \(p=2\)  to have channels 64, 32 and 16 from 1024, 512 and 256, respectively. We implemented our work with PyTorch [6]}. We used learning rate of 5e-4. We used augmentation introduced in [7]} for training as shown in Table REF . We adopted early-stopping for all the training, which we picked the model with best validation score. We set the batch size as 16.


Components of VAT.
For the ablation study of components of VAT, we defined the baseline model which substitutes VEM with 4D max-pooling. As VTM is a transformer module that takes an input and outputs the same shape to input, it can simply be skipped. Residual connections are simply removed to test its efficiency, and appearance affinity is excluded at the decoder. This experiment is conducted in a sequential manner, where each component is added progressively.


Different Cost Aggregators.
For the ablation study of cost aggregators, we used codes released by authors of works, including standard transformer [8]}, center-pivot 4D convolutions [9]}, linear transformer [10]}, and fastformer [11]}. We simply replaced VTM with those in order to make a fair comparison.


Different backbone networks.
Ablation study on different feature backbone follows similar procedure, as PVT [12]} and swin transformer [13]} have pyramidal architecture that outputs features of size and dimensions similar to ResNet [4]}.

<TABLE><TABLE><TABLE><TABLE>
VAT in Semantic Correspondence.
For the experiments to validate effectiveness of VAT on semantic correspondence, we used the three most widely used datasets in this task, which include SPair-71k [1]}, PF-PASCAL [2]} and PF-WILLOW [3]}. We made minor modifications to our model: we excluded bilinear upsampling in the affinity-aware decoder, which the original output shape is \(128 \times 128\)  but with the modification, now the output shape is \(32 \times 32\)  as two x2 bilinear upsampling is excluded. We used embedded features from conv4\(\_\)  and conv5\(\_\) x, which is different from the training setting for the experiments on few-shot segmentation datasets. Note that we do not utilize segmentation mask for this task. We only use keypoints provided in the datasets for training similarly to [18]}. We used the same data augmentation scheme [18]}, [7]} as Table REF . We used learning rate of 3e-6 for the feature backbone and 3e-4 for VAT. We used batch size of 8. For the evaluation, we followed the standard protocol [21]}, [22]}, [23]}, [24]}, [18]}, [26]}.
<TABLE>
Appendix B. Failure Cases
Overall, we observe that our method may lack an ability to preserve fine-details and address multi-correspondence as shown in Figure REF . Given an object in the support image and multiple corresponding objects in query image, VAT sometimes seem to fail finding correspondence. From these, perhaps, it could be argued that the proposed method is better at encoding matching information and effectively aggregating them than others, but it might lack an ability to better find multi-objects or preserve the fine-details. Better means to consider both factors either by introducing some confidence module [27]} or preserve fine-details when increasing the resolutions of the predicted mask during the process in decoder can be a promising direction future work.

Appendix C. Memory and Run-time
We additionally provide memory and run-time comparison to other aggregators and HSNet [9]}. The results are obtained using a single NVIDIA GeForce RTX 3090 GPU and Intel Core i7-10700 CPU. As shown in Table REF , we observe that VAT is relatively slower and comsumes more memory than other aggregators. About 0.3 GB of more memory consumption and 5 ms slower run-time occur with the proposed VAT in return for the performance. From the perspective of aggregators, 0.3 GB and 5 ms gap seem quite trivial to us.
We also provide direct comparison to the current state-of-the-art method, HSNet [9]}, in Table REF . We observe that memory consumption gap is quite large, which is an apparent limitation to the proposed method. Run-time gap is quite trivial, enabling real-time inference. Note that standard transformer can not be used due to OOM. In light of this, although we managed to propose a method to aggregate a raw correlation map without arbitrarily changing its resolutions prior to feeding into networks, we observe relatively large memory consumption gap to HSNet [9]}, which suggests a promising direction for future work, i.e., even more efficient method while enabling to process at higher spatial resolutions.
<FIGURE>
Appendix D. More Results
For few-shot segmentation, VAT clearly sets new state-of-the-art for all the benchmarks, demonstrating its effectiveness. Also as VAT outperforms other SOTA methods by large margin for semantic correspondence, we argue that the cost aggregation is indeed a prime importance, even for few-shot segmentation as well. We argue that this performance boost in few-shot segmentation can be attributed to VAT's ability to find accurate semantic correspondences when tackling the segmentation task. In light of this, we are suggesting a paradigm shift for the few-shot segmentation task.

Quantitative Results on Semantic Correspondence.
As shown in Table REF  and Table REF , we provide more quantitative results on SPair-71k [1]}, PF-PASCAL [2]}, and PF-WILLOW [3]} in comparison to other semantic correspondence methods, including CNNGeo [34]}, WeakAlign [35]}, NC-Net [36]}, HPF [21]}, SFNet [38]}, DCC-Net [39]}, GSF [24]}, SCOT [23]}, DHPF [22]}, CHM [26]}, MMNet [44]}, PMNC [45]} and CATs [18]}.


Qualitative Results.
As shown in Figure REF , Figure REF , Figure REF , Figure REF  and Figure REF , we provide qualitative results on all the benchmarks, which includes PASCAL-5\(^{i}\)  [47]}, COCO-20\(^{i}\)  [48]}, FSS-1000 [49]}, PF-PASCAL [2]}, PF-WILLOW [3]} and SPair-71k [1]}.
<FIGURE><FIGURE><FIGURE><FIGURE><FIGURE>