Text-guided image manipulation approaches have been most commonly explored in the computer vision community. Several studies utilized a GAN-based encoder-decoder architecture to preserve the visual features in the manipulation process driven by descriptive text inputs [1]}, [2]}, [3]}. Recently, StyleCLIP [4]} utilized the latent space of the pre-trained StyleGAN and a CLIP [5]}-based text-image joint embedding space to synthesize visually more plausible images [4]}. Further, TediGAN [7]} also explored using a GAN inversion technique based on multi-modal mapping. These approaches generate manipulated images by minimizing the CLIP distance between an input text and an image generated by a StyleGAN-based generator. Similarly, in this work, we also follow this strategy to optimize distances in the CLIP embedding space, but we extend the CLIP space into an image-text-sound joint embedding space, improving its representational power for the image manipulation task. We empirically observe that a sound can convey a more complex semantic context in a scene (e.g., temporal context, music genre, tone, and volume), producing a more visually plausible image manipulation.
<FIGURE>