On Certifying and Improving Generalization to Unseen Domains
Domain Generalization (DG) aims to learn models whose performance remains
high on unseen domains encountered at test-time by using data from multiple
related source domains. Many existing DG algorithms reduce the divergence
between source distributions in a representation space to potentially align the
unseen domain close to the sources. This is motivated by the analysis that
explains generalization to unseen domains using distributional distance (such
as the Wasserstein distance) to the sources. However, due to the openness of
the DG objective, it is challenging to evaluate DG algorithms comprehensively
using a few benchmark datasets. In particular, we demonstrate that the accuracy
of the models trained with DG methods varies significantly across unseen
domains, generated from popular benchmark datasets. This highlights that the
performance of DG methods on a few benchmark datasets may not be representative
of their performance on unseen domains in the wild. To overcome this roadblock,
we propose a universal certification framework based on distributionally robust
optimization (DRO) that can efficiently certify the worst-case performance of
any DG method. This enables a data-independent evaluation of a DG method
complementary to the empirical evaluations on benchmark datasets. Furthermore,
we propose a training algorithm that can be used with any DG method to provably
improve their certified performance. Our empirical evaluation demonstrates the
effectiveness of our method at significantly improving the worst-case loss
(i.e., reducing the risk of failure of these models in the wild) without
incurring a significant performance drop on benchmark datasets.