Cross-modal/multi-modal pre-training methods have been proposed: OpenL3[1]} was pre-trained by using audio-visual correspondence as training signal. Wang et al. [2]} was pre-trained by using correspondence between video, spectrograms, and raw waveforms to learn general-purpose audio representations.
COALA[3]} was pre-trained by aligning the learned latent representations of audio and associated tags and evaluated on SER and music tasks.
