In this paper, we propose a novel pretext task, namely Cross-Modal Attention Consistency (CMAC), for exploring the bidirectional local correspondence property between visual and acoustic signals.
The core insight is to make the visual encoder attend to regions where sounds are made and the audio encoder attend to sound frequencies from interested objects, as shown in Fig. REF .
To achieve the goal, CMAC proposes to align the regional attention generated purely from the visual signal with the target attention generated under the guidance of acoustic signal, and do a similar alignment for frequency grounding on the acoustic attention.
Compared with traditional cross-modal contrastive manner [1]}, [2]}, CMAC provides a novel mechanism that considers bidirectional local correspondence between spatio-temporal visual clues and audio-spectrogram signals via attention consistency.
