To learn these joint latent representations, we use a typical contrastive learning approach that follows the work by Radford et al. [1]}: the latent representations for a positive triplet pair are mapped close together in the embedding space while representations of negative pair samples are mapped farther away from each other. Specifically, we use InfoNCE loss [2]} to map positive audio-text pairs and audio-visual pairs close together in the CLIP-based joint embedding space, and we map negative pairs farther away from each other.
Formally, given a minibatch of \(N\)  audio-text representation pairs \(\lbrace {\bf {a}}_i, {\bf {t}}_j\rbrace \)  for \(i\in \lbrace 1, 2, \dots , N\rbrace \) , we first compute the \((i,j)\)  component of the audio-text similarity matrix \(M^{a\rightarrow t} \in \mathbb {R}^{N \times N}\)  between the \({\bf {a}}_i\)  and \({\bf {t}}_j\)  latent representation as:
\(M^{a\rightarrow t}_{(i, j)}=\cfrac{\exp ({\bf {a}}_i \cdot {\bf {t}}_j/\tau ) }{\sum _{\textnormal {k=1}}^N\exp ({\bf {a}}_i \cdot {\bf {t}}_k/\tau )},\) 
