A Closer Look at Smoothness in Domain Adversarial Training
Domain adversarial training has been ubiquitous for achieving invariant
representations and is used widely for various domain adaptation tasks. In
recent times, methods converging to smooth optima have shown improved
generalization for supervised learning tasks like classification. In this work,
we analyze the effect of smoothness enhancing formulations on domain
adversarial training, the objective of which is a combination of task loss (eg.
classification, regression, etc.) and adversarial terms. We find that
converging to a smooth minima with respect to (w.r.t.) task loss stabilizes the
adversarial training leading to better performance on target domain. In
contrast to task loss, our analysis shows that converging to smooth minima
w.r.t. adversarial loss leads to sub-optimal generalization on the target
domain. Based on the analysis, we introduce the Smooth Domain Adversarial
Training (SDAT) procedure, which effectively enhances the performance of
existing domain adversarial methods for both classification and object
detection tasks. Our analysis also provides insight into the extensive usage of
SGD over Adam in the community for domain adversarial training.