Self-supervised learning (SSL) methods are proposed to learn discriminative features from large-scale unlabeled data without any human-annotated labels [1]}.
One branch is developed from Momentum Contrast (MoCo) [2]} by treating a pair of augmentations of a sample as a positive pair and all other samples as negative pairs. Since the number of negative samples greatly affects the final performance, MoCo series [3]}, [2]} require large batches or memory banks. Among them, MoCoV3 [3]} is a Transformer-specific version. Fu  [6]} have verified ResNet50 can be well pre-trained by a modified MoCoV2 on human images in person ReID. Many recent works have shown models can learn feature representation without discriminating between images. In this direction, Ge  [7]} propose a new paradigm called BYOL, where the online network predicts the representation of the target network on the same image under a different augmented view. Large batch size is unnecessary in BYOL since negative samples are not needed. Many variants have successfully improved BYOL in various ways. One of them is DINO [8]}, where a centering and sharpening of the momentum teacher outputs is used to avoid model collapse. DINO achieves state-of-the-art performance with ViTs on both ImageNet classification and down-stream tasks. Xie combines MoCo with BOYL to propose a Transformer-specific method called MoBY. Given that there have been more SSL methods designed specifically for Transformers, we will focus on several of the state-of-the-art options in the following experiments, MoCo series, MoBY and DINO.
