Suppose we have an image dataset with \(n\)  instances sampled i.i.d. from input space \(\mathcal {X} \)  is denoted as \(\mathcal {D}= \lbrace x_1, x_2, \dots , x_n\rbrace \) , we can obtain the embeddings of these images as \(\mathcal {U} = \lbrace \mathbf {u}_1, \mathbf {u}_2, \dots , \mathbf {u}_n\rbrace \)  where \(\mathbf {u}_i=g(x_i)\)  is obtained via the image encoder \(g(.)\)  of CLIP. To capture the semantic meaning of these images, we introduce a semantic dataset \(\mathcal {T} = \lbrace t_1, t_2, \dots , t_m\rbrace \)  that includes \(m\)  noun phrases from WordNet [1]} and define a function \(h(t_i)\)  to obtain the embedding of each noun \(t_i\)  from CLIP [2]}, by constructing a sentence like “\(\texttt {A photo of a } \lbrace t_i\rbrace \) ” and obtain embeddings as \(\mathcal {V} = \lbrace \mathbf {v}_1, \mathbf {v}_2, \dots , \mathbf {v}_m\rbrace \)  where \(\mathbf {v}_i=h(t_i)\)  from the text encoder of CLIP. Let \(c\)  be the number of categories, our goal is to group the images in \(\mathcal {D}\)  into \(c\)  clusters with the help of CLIP model. Let \(f(g(\mathcal {D});\phi ):\mathcal {V}\rightarrow \mathbb {R}^c\)  denotes the
network with parameters \(\phi \)  that maps an image \(x_i\)  with embedding \(\mathbf {u}_{i}\)  into soft cluster assignment probabilitiy \(\mathbf {q}_{i}\) . \(f\)  is implemented by a multilayer perceptron (MLP). Notably, the image and text encoders in CLIP are kept frozen during the training process, i.e., the parameters in the functions \(g(.)\)  and \(h(.)\)  are fixed.
