Dataset and Models Our method is validated on the popular ImageNet 1k dataset [1]}. This dataset contains 1.28M images in the training set and 5K images in the validation set from 1000 classes. We only use the training set during the process of self-supervised learning. As to models, we choose the classical DeiT-S [2]} and popular Swin-T [3]} as representatives of all transformer-based architectures. After the backbone, a 3-layer MLP with hidden dimension 2048 is added as the projection head. When evaluating our pretrained model, we both use the k-NN algorithm and train a linear classification for 100 epochs as former works. Top-1 accuracy is reported.
