is an important ingredient of self-supervised learning, and multiple works have studied its impact on the transfer performance of models.
While it is possible to learn high quality features from non-curated (eg. YFCC or IG) data using instance discrimination, this usually requires order of magnitude more data than ImageNet [1]}, [2]}.
Similarly, one can perform supervised pre-training using weakly supervised data, such as using hashtags as labels, but this strategy also requires large amount of data to work well [3]}, [4]}, [5]}.
On the other hand, it was shown that for many natural language processing tasks, increasing the size of the pre-training dataset did not lead to strong improvement when using denoising autoencoders [6]}.
Finally, some work studied how much could be learned from a single pre-training image [7]} or from synthetic data [8]}, [9]}.
