There is a general class of methods called self-supervised learning that focuses on learning rich representation exclusively from unlabeled data. The idea is to use inherent biases in natural images to design a pseudo task for which the annotation is automatic. Then, by forcing the model to solve the pseudo task, the model will learn rich features that capture the manifold of natural images. For instance, since most natural images are upright, we, humans, can detect if an image is rotated, so in RotNet [1]}, we can randomly rotate the images and train a model to predict the applied rotation angle. This pseudo-task does not need any annotations and learns rich visual features.
