Gated Domain Units for Multi-source Domain Generalization
Distribution shift (DS) is a common problem that deteriorates the performance
of learning machines. To overcome this problem, we postulate that real-world
distributions are composed of elementary distributions that remain invariant
across different domains. We call this an invariant elementary distribution
(I.E.D.) assumption. This invariance thus enables knowledge transfer to unseen
domains. To exploit this assumption in domain generalization (DG), we developed
a modular neural network layer that consists of Gated Domain Units (GDUs). Each
GDU learns an embedding of an individual elementary domain that allows us to
encode the domain similarities during the training. During inference, the GDUs
compute similarities between an observation and each of the corresponding
elementary distributions which are then used to form a weighted ensemble of
learning machines. Because our layer is trained with backpropagation, it can be
easily integrated into existing deep learning frameworks. Our evaluation on
Digits5, ECG, Camelyon17, iWildCam, and FMoW shows a significant improvement in
the performance on out-of-training target domains without any access to data
from the target domains. This finding supports the validity of the I.E.D.
assumption in real-world data distributions.