AT and Its Variants. Szegedy et al. (2014) reported that one can inject imperceptible perturbations to fool deep models. Among the proposed defenses, Adversarial Training (AT) (Goodfellow, Shlens, and Szegedy 2015;Madry et al. 2018; is the promising and representative approach to training robust models (Athalye, Carlini, and Wagner 2018). Recently, Salman et al. (2020) showed that adversarially learned features could transfer better than standardly trained models, while various works (Volpi et al. 2018;Sinha, Namkoong, and Duchi 2018;Shankar et al. 2018;Ford et al. 2019;Qiao, Zhao, and Peng 2020;Yi et al. 2021;Gokhale et al. 2021) adopt samplewise adversarial training or adversarial **data** augmentation to improve OOD robustness. However, most discussions are limited to distributional robustness w.r.t. Wasserstein distance. The small perturbations used in AT make it less practical to account for real-world OOD scenarios (e.g., correlation to backgrounds), thus  incorporated low-rank structured priors into AT for this kind of large **data** distribution shifts. Moreover, previous work shows that there seems to be a trade-off between the two distribution shifts: an algorithm that performs well on one task tends to perform poorly on the other (Ye et al. 2022). Instead, in this work, our proposed method can achieve fair performance on both correlation shift and diversity shift tasks.