Here we omit the projection heads for simplicity.
\(\mathcal {L}_{cl}\)  is a variant of contrastive loss, where the main difference between \(\mathcal {L}_{cl}\)  and \(\mathcal {L}_{nce}\)  is that we include within-modal negative pairs \((\kappa _{n}^{v}, \kappa _{m}^{v})\)  in Eq. (REF ) to introduce within-modal discrimination knowledge, which is ignored by most of the existing methods [1]}, [2]}, [3]} as in Eq. (REF ).
AVID [4]} also considers the within-modal knowledge by adding \(\mathcal {L}_{nce}(v_{n1},v_{n2})\)  and \(\mathcal {L}_{nce}(a_{n1},a_{n2})\)  to Eq. (REF ), which, however, requires saving two clips for \(n\) -th video in a mini-batch as positive pairs.
In contrast, our simple yet effective modification requires no extra positive samples and no additional memory overhead.
