Cross-modal unsupervised learning. Compared to images, videos are more commonly seen and contain a rich variety of modalities, e.g. visual, audio, and speech signals.
The correlation between different modalities is a natural supervision that has attracted increasing interest [1]}.
Some works [2]}, [3]}, [4]}, [5]} leverage the speech as a weak supervision, and other works explore the audio self-supervision to boost visual localization [6]}, audio-visual source separation [7]}, [8]}, [9]}, [10]}, and representation learning [11]}, [12]}, [13]}, [14]}.
In this work, we focus on visual-audio unsupervised learning, which aims at jointly learning visual and audio representations in an unsupervised manner.
A general paradigm is to detect whether the visual and audio signals are from the same video, which is commonly solved via cross-modal contrastive learning [15]}, [16]}, [17]}, [18]}.
AVTS [15]} further employs the temporal synchronization by defining positive visual-audio pairs only when they are temporally synchronized, constraining the representations to understand the temporal content.
In addition to the cross-modal learning, AVID [16]} introduces the within-modal information via extra visual-visual and audio-audio contrastive learning, but it requires doubling the memory cost for both within- and cross-modal positive pairs.
GDT [17]} explores more comprehensive data augmentations and data pair constructions.
Different from cross-modal contrastive learning, XDC [22]} leverages unsupervised clustering in one modality to supervise representations of the other modality. AVSlowFast [18]} proposes a visual-audio encode architecture that contains slow and fast visual pathways to explore knowledge at different temporal scales.
Though above methods achieve promising results, they only consider cross-modal instance discrimination by distinguishing different clips, while ignoring the local correspondence between spatio-temporal regions and acoustic frequencies.
