Modern deep learning techniques have achieved remarkable success in many tasks (He et al. 2016;Wang et al. 2017;Brown et al. 2020). However, deep models will suffer catastrophic performance degradation under some scenarios, as they tend to exploit spurious correlations in the training **data** (Beery, Van Horn, and Perona 2018). One of those representative scenarios is the Out-of-Distribution (OOD) generalization, where the trained model is expected to perform well at the test time, even when the training and testing **data** come from different distributions (Zhang et al. 2021a). Another representative scenario in which deep models are unstable is the adversarial example. Researchers have found that deep models are quite brittle, as one can inject imperceptible perturbations into the input and cause the model to make wrong predictions with extremely high confidence (Szegedy et al. 2014).