Most works on unsupervised learning of features learn the models on supervised datasets like ImageNet [1]}.
Some previous works have explored unsupervised training on images [2]}, [3]}, [4]} and videos [5]} taken “in the wild”.
The conclusions of these works were mixed but these studies were conducted at a relatively small scale, both in model and data size.
There are now evidences that self-supervised pretraining benefits greatly from large models [6]}, [7]}, [8]}, [9]}.
Our work builds upon these findings to explore if we can learn good visual representations by training significantly larger models on random, uncurated and unlabeled images.
