Unsupervised feature learning has a long history in computer vision, and many approaches have been explored in this space.
Initially, methods using a reconstruction loss have been explored with the use of autoencoders [1]}, [2]}.
More recently, a similar paradigm has been used in the context of masked-patch-prediction models [3]}, [4]}, [5]}, showing that scalable pre-training can be achieved.
Alternatively, many creative pretext tasks have also been proposed, showing that good features can be trained that way [6]}, [7]}, [8]}, [9]}, [10]}, [11]}, [12]}, [13]}, [14]}, [15]}, [16]}, [17]}, [18]}.
A popular trend was using instance discrimination [19]}, [20]}, [21]}, [22]}, [23]}, [24]}, [25]} as a training task.
In this setup, each sample in the dataset is also it's own class.
Several other interesting papers proposed to learn joint embeddings, “pulling together” different views of the same image [26]}, [27]}, [28]}.
Finally, a large body of work considered grouping instances and using clustering [29]}, [30]}, [31]}, [32]}, [33]}, [34]}, [35]}, [36]}, [37]} or soft versions thereof [38]}, [39]} as training tasks.
Many of those works have shown excellent performance on numerous downstream tasks, often showing that unsupervised features can surpass supervised ones.
In this paper, we use the model proposed by Caron  [38]}, using soft assignments of images to prototypes.
