Here we explore which kind of within-modal information really works in cross-modal contrastive learning.
Two types of sampling policies are considered: a) Within-modal negative sampling that takes two samples (with same modality) from different videos to form a negative pair and distinguishes them as in Eq. (REF ); b) Within-modal positive sampling that takes two samples (with same modality) from the same video to form a positive pair and aligns them.
Note that using both policies actually forms the strategy in AVID [1]}.
The results are shown in Table REF .
We observe that the negative sampling could benefit CMAC a lot on both datasets without extra memory expense, since we can simply utilize the samples from previous batch (stored in memory bank) to form negative pairs.
In contrast, the positive sampling harms the performance and even costs twice as much memory as negative sampling, due to that it requires saving two clips for each video.
The possible reason for the performance drop is that aligning within-modal positive pairs would discard the intrinsic temporal synchronization in each video.
