It is important note that the models were trained with labels in all above-mentioned methods, but biases might still be learned when using self-supervised training schemes. In [1]}, Sirotkin et al. looked for the presence of bias in representations learned using state-of-the-art self-supervised learning (SSL) procedures. In particular, they pre-trained models on ImageNet [2]} using a variety of SSL techniques and showed that there was a correlation between the type of model and the number of incorporated biases.
Thus, they demonstrated that biases can be learned even without supervision. In a Meta-Learning fashion, [3]} also propose to learn how to split a dataset such that predictors learned on a training split which cannot generalize on the test split.
