Several follow-up works comment on the performance of the invariant risk framework, and compare against standard Empirical Risk Minimization (ERM). It has been noted that carefully tuned ERM outperforms state-of-the-art domain generalization approaches, including IRM, across multiple benchmarks [1]}. The failure of IRM may stem from the gap between the proposed framework and its practical “linear” version (IRMv1), which fails to capture natural invariances [2]}. The authors of [3]} demonstrate that a near-optimal solution to the IRMv1 objective, which nearly matches IRM on training environments, does no better than ERM on environments that differ significantly from the training distribution. The authors of [4]} argue that IRM has an advantage over ERM only when the support of the different environment distributions have a significant “overlap”. However, unlike previous work which aim to minimize OOD risk, our primary focus is to understand the trade-off between risk minimization and fairness across environments in the setting when such a significant “overlap” may not exist. Consequently, we analyze how IRM behaves in these settings and explore alternatives via the notion of partial invariance. The notion of fairness considered in this paper relates to group sufficiency [5]}, which is intricately linked with the IRM objective. In fact, several works build upon the IRM framework by explicitly minimizing a quantity resembling this notion of fairness for achieving invariance [6]}, [7]}.
