Previous sound-to-visual synthesis works [1]}, [2]}, [3]}, [4]}, [5]}, [6]} have attempted to convert sound signals into realistic visual content. However, these works have a limitation in creating or changing intuitive image styles with various audio domains, as they do not leverage pre-trained joint multi-modal representations like CLIP [7]}.
To solve this problem, many recent studies try to match the sound embedding space with the CLIP embedding space [8]}, [9]}, [10]}.
In particular, Lee et al. [9]} proposed a method to provide sound-based guidance by matching the sound embedding space with the CLIP embedding space. After matching the multimodal embedding space, the StyleGAN source latent vector moves under CLIP-based guidance by minimizing the cosine distance of the manipulated image embedding and sound embedding.
This approach allows the StyleGAN source latent vector in \(\mathcal {W}+\)  space to obtain a robust direction for semantic cues while being intensity-aware for various sound inputs.
<FIGURE><FIGURE>