Despite the impressive performance of the above methods, their pretext supervisions only consider the instance level relationship between global modality representations, e.gthe aggregated spatio-temporal features and audio spectrogram features [1]}, [2]}.
This inevitably discards fine-grained local clues, i.ethe relation between each spatio-temporal region and foreground sound frequency.
In the human sensory system, our visual perception is sensitive to the spatial regions where sounds are made, and vice versa our auditory perception is sensitive to the sound frequencies belonging to foreground objects.
This phenomenon, called bidirectional local correspondence, is natural and intuitive to associate spatio-temporal visual clues with acoustic frequencies.
[rgb]0,0,0Some early works [3]}, [4]}, [5]} study the global alignment between modalities for representation learning, while ignoring such local relation.
One recent exploration [6]} considers it in image feature learning via co-clustering. However, their method has not been validated in the challenging large-scale video unsupervised learning task.
