Unsupervised learning from images. Due to unavailable human annotation, existing methods focus on designing pretext tasks [1]}, [2]}, [3]}, [4]} to learn good data representations.
For example, a seminal work [5]} augments each image into several transformations and regards each image as an exemplar, thus a traditional classification task can be defined as recognizing image transformations into their exemplar category.
Besides, CFN [6]} designs a jigsaw puzzle task by predicting orders of different patches from the same image, and RotNet [4]} predicts the rotation angles of an image.
These proxy tasks aim to understand some low-level image concepts, such as rotation invariance and spatial relationship among image patches.
Recently, contrastive loss [8]}, [9]}, [10]}, [11]}, [12]}, [13]}, [14]}, [15]} has attracted great attention, which makes different transformations from the same image attract and those from different images repel.
Compared to fixed prototypes in exemplar methods, the contrastive loss can vary on-the-fly training.
Two representative methods are MoCo [16]} and SimCLR [17]}.
MoCo designs a dynamic dictionary with a moving-averaged encoder to store a large-scale and consistent instance representation set, and SimCLR explores the importance of projection head and comprehensive data augmentations in unsupervised learning.
<FIGURE>