
Powerful image generative models.
Such models include recent generative adversarial networks (GANs) [1]}, [2]}, [3]}, variational autoencoders [4]} and diffusion models [5]}, [6]}, [7]}, [8]}, that can produce images with high fidelity and diversity.
Formally, this process can be denoted as \(x=G(z)\)  where the generative model \(G: \mathbb {R}^{d_z} \rightarrow \mathbb {R}^{h \times w \times 3}\)  converts a latent space variable \(z \in \mathbb {R}^{d_z}\)  to an RGB image of height \(h\) , weight \(w\)  and 3 color channels. \(x \in \mathbb {R}^{h \times w \times 3}\) .
\(z\)  could be further manipulated to allow generating more suitable \(x\)  [9]}, allowing artist to control the generation of artworks that fall in desired genres [10]}.

Joint modeling of images and natural language.
This idea has been long in the making [11]}, [12]}, but only recently given a convincing implementation thanks to progress in natural languages modeling [13]}, [14]}, and notably the ability to embed jointly images and text so well that the need for task-specific fine-tuning is eliminated, as shown in CLIP [15]}.
CLIP provides two jointly-trained, differentiable encoders, \(E_{\mathrm {I}}: \mathbb {R}^{h \times w \times 3} \rightarrow \mathbb {R}^d\)  and \(E_{\mathrm {T}}: \mathcal {T} \rightarrow \mathbb {R}^d\) , for image and text respectively.
We do not further elaborate the domain of text \(\mathcal {T}\)  as it is not the focus of this work.
Formally, given an image \(x\)  and a text \(t\) , and a distance function
\(D: \mathbb {R}^{d} \times \mathbb {R}^d \rightarrow \mathbb {R}_{+}\)  the encoded image \(u=E_{\mathrm {I}}(x)\)  and the encoded text \(v=E_{\mathrm {T}}(t)\)  are in a common comparable space \(\mathcal {U} = \mathbb {R}^d\) , and \(D(u, v)\)  measures the similarity between \(x\)  and \(t\) .
In the case of CLIP that is trained with cosine distance,
practically \(D\)  could be chosen as cosine distance or geodesic distance, both effectively measuring the angle between the two vectors and being trivially differentiable.
Ideally, text-driven image generation is now feasible by iteratively adjusting the latent space vector \(z\) ,
to minimize \(D(u, v)\) , the distance between the encoded image \(x=G(z)\)  and encoded user-specific prompt \(t\) .
As \(G\) , \(E_{\mathrm {I}}\)  and \(D\)  are differentiable, \(z\)  could be updated using gradient Descent:
\(z \leftarrow z - \gamma \nabla _{z} F(z)\) 
where \(\nabla _{z} F\)  is the gradient of \(F\)  defined as \(F(z) = D(E_{\mathrm {I}}(G(z)), E_{\mathrm {T}}(t))\)  and \(\gamma \)  is a learning rate.

