To overcome this problem, recent studies [1]}, [2]}, [3]} align the sound embedding space with the CLIP embedding space. In order to predict the same CLIP image representations from the audio stream, Wav2CLIP [1]} freezes the CLIP image encoder and trains the audio encoder on the audio streams from videos. AudioCLIP [2]} extends modality and scale by using a newly collected trimodal audio-visual-text dataset. Lee et al. [3]} trains an audio encoder from scratch with a audio-text dataset and audio-visual streams from video to obtain a CLIP-based sound latent representation.
