Self-Supervision on Images and Text Reduces Reliance on Visual Shortcut Features
Deep learning models trained in a fully supervised manner have been shown to
rely on so-called "shortcut" features. Shortcut features are inputs that are
associated with the outcome of interest in the training data, but are either no
longer associated or not present in testing or deployment settings. Here we
provide experiments that show recent self-supervised models trained on images
and text provide more robust image representations and reduce the model's
reliance on visual shortcut features on a realistic medical imaging example.
Additionally, we find that these self-supervised models "forget" shortcut
features more quickly than fully supervised ones when fine-tuned on labeled
data. Though not a complete solution, our experiments provide compelling
evidence that self-supervised models trained on images and text provide some
resilience to visual shortcut features.