For representation learning, we used the CLIP pre-training model, whose visual and text backbones are ViT-32 [1]} and Transformer [2]}, separately. We obtained features from the image encoder of CLIP and then trained a cluster head. The cluster head is a fully connected layer with a size of \(d\times c\) , where \(d\)  and \(c\)  are the pre-training feature dimension and the number of clusters respectively. During the training, the epoch numbers of all datasets were set to 100 with a batch size of 128. Considered the training process of Imagenet-Dogs and Imagenet-tiny fluctuates, their early stop number were set to 10. Before training, all datasets were augmented with the same method used in CLIP [3]}, i.e., a random square crop from resized images. The nearest neighbors were searched through Faiss Library  [4]}. The best hyper-parameters used for five benchmark datasets are shown in Table REF .
<TABLE>