In standard machine learning frameworks, models can be expected to generalize well to unseen data if the data follows the same distribution [1]}. However, it has been noted that distribution shifts during test time (due to data being from different sources, locations or times) can severely degrade model performance [2]}, [3]}. For instance, in a computer vision task to classify camels and cows, the authors ([4]}) showed that during testing, the model with perfect training loss misclassified cows as camels during test time, if the image background was that of a desert. The issue arose due to the model picking up a strong but spurious correlation: during training, most cow images had green pastures, whereas camel images had deserts. Consequently, the model began to associate cows to green pastures and camels to deserts, leading to misclassification during test time. The key observation is that the vanilla classifiers are often prone to exploiting such spurious features during training (in this case, the image background), which can then hamper its performance in out-of-distribution (OOD) tasks. To fix this problem, multiple lines of research have explored the idea of enforcing invariance during learning [5]}, [6]}, [7]}, [8]}, to find invariant, causal, features that are independent of the input distribution which may vary across training environments [9]}. With respect to the vision task considered before, the features specific to the cow in the image (color, shape, etc.) would correspond to invariant features. Therefore, we can obtain a robust predictor by capturing only invariant features, which inevitably capture some causal relationships about the data and thus generalize well in OOD tasks.
