These limitations have motivated the development of self-supervised pre-training methods which learn directly from data, without relying on annotations. Most notably, the contrastive and joint embedding approaches [1]}, [2]}, [3]}, [4]}, [5]} can serve as effective pre-training strategies. While obtaining a strong performance on numerous tasks, such methods have a strong bias towards ImageNet data since the transformations have been hand-designed to perform well on the ImageNet benchmark. Some of the most effective transformations, like cropping, rely on the images being object centric [6]}.
When applied on uncurated data, these methods degrade significantly and require larger datasets to obtain similar performance [7]}.
