In self-supervised representation learning, labels are mined from the data itself to achieve generalization beyond that from human annotations, especially when the training data is at scale.
Past works demonstrate generalization through k-nearest-neighbors [1]} and zero-shot classification on the learned features [2]}, or finetuning the model for a limited schedule [3]}, [4]}.
The central problem under investigation is how to extract the training labels automatically and formulate the pretext tasks. In high-level vision, such pretext tasks include predicting colors from a grayscale image [5]}, inpainting pixels given the spatial context [6]} or through autoregression [7]}, predicting the orientation of a rotated image [8]}, solving a jigsaw puzzle given shuffled image patches [9]}, and others [10]}, [11]}.
The key idea is that the network has to learn semantics in order to solve the pretext tasks.
Recently, there has been a resurgence of the context prediction pretext task [12]} that has accompanied the rise of visual transformers [13]}. Input visual tokens are masked, and the model is trained to predict the masked tokens from the visible tokens in a BERT fashion. The target tokens could be represented by dVAE tokens [12]}, raw pixel values [4]}, or features [16]} from an online learned encoder [17]}, [18]}, [19]}, [20]}. Such representations are shown to surpass prior art when finetuned for downstream tasks [4]}.
