Following MLM paradigm in NLP [1]}, [2]}, iGPT [3]} trains self-supervised Transformers by masking and reconstructing pixels, while ViT [4]} masks and reconstructs patches. Recently, the most competitive pretext task for self-supervised visual representation learning is instance discrimination [5]}, [6]}, [7]}, [8]}, [9]}, [10]}. The learning objective is simply to learn representations by distinguishing each image from others, and this approach is quite intractable for large-scale datasets.
MoCo [5]} improves the training of instance discrimination methods by storing representations from a momentum encoder instead of the trained network.
SimCLR [6]} shows that the memory bank can be entirely replaced with the elements from the same batch if the batch is large enough. In order to avoid comparing every pair of images and incur overfitting, BYOL [9]} directly bootstraps the representations by attracting the different features from the same instance. SwAV [10]} maps the image features to a set of trainable prototype vectors and proposes multi-crop data augmentation for self-supervised learning to increase the number of views of an image. MoCov3 [15]} and DINO [16]} apply the self-supervised learning methods of computer vision to Transformers and achieve superior performance in image classification task.
These works achieve comparable results compared to supervised ImageNet [17]} pre-training. The success of these methods suggest that it is of central importance to learn invariant features by matching positive samples. However, almost all of these self-supervised learning methods formulate the learning process as image-level prediction using global features, so they lack the ability to pay attention to local features.
