This is in contrast with natural language processing, where nowadays, most applications use large models which were pre-trained on uncurated data.
In particular, the (masked) language modeling loss has been applied to transformer networks, leading to the BERT model [1]}, which is now the foundation of most NLP models.
Inspired by this success, [2]} have shown the potential of the mim task to pre-train vision transformers. Such a model can be thought of as a denoising autoencoder [3]} where the noise corresponds to the patch masking operation.
This technique has been successfully applied to ImageNet, but research questions remain:
