Unsupervised domain adaptation aims to generalize the hypothesis trained in a source domain to an unlabeled target domain. One popular approach to this problem is to learn domain-invariant embeddings for both domains. In this work, we study, theoretically and empirically, the effect of the embedding complexity on generalization to the target domain. In particular, this complexity affects an upper bound on the target risk; this is reflected in experiments, too. Next, we specify our theoretical framework to multilayer neural networks. As a result, we develop a strategy that mitigates sensitivity to the embedding complexity, and empirically achieves performance on par with or better than the best layer-dependent complexity tradeoff. Domain adaptation is critical in many applications where collecting large-scale supervised data is prohibitively expensive or intractable, or where conditions at prediction time can change. For instance, self-driving cars must be robust to different weather, change of landscape and traffic. In such cases, the model learned from limited source data should ideally generalize to different target domains. Specifically, unsupervised domain adaptation aims to transfer knowledge learned from a labeled source domain to similar but completely unlabeled target domains. One popular approach to unsupervised domain adaptation is to learn domain-invariant representations (Ben-David et al., 2007; Long et al., 2015; Ganin et al., 2016) , by minimizing a divergence between the representations of source and target domains. The prediction function is learned on these "aligned" representations with the aim of making it domain-independent. A series of theoretical works justifies this idea (Ben-David et al., 2007; Mansour et al., 2009; Ben-David et al., 2010; Cortes & Mohri, 2011) . Despite the empirical success of domain-invariant representations, exactly matching the representations of source and target distribution can sometimes fail to achieve domain adaptation. For example, Wu et al. (2019) show that exact matching may increase target error if label distributions are different between source and target domain, and propose a new divergence metric to overcome this limitation. Zhao et al. (2019) establish lower and upper bounds on the risk when label distributions between source and target domains differ. Johansson et al. (2019) point out the information lost in non-invertible embeddings, and propose different generalization bounds based on the overlap of the supports of source and target distribution. In contrast to previous analyses that focus on changes in the label distributions or joint support, we study the effect of embedding complexity. In particular, we show a general bound on the target risk that reflects a tradeoff between embedding complexity and the divergence of source and target domains. A too powerful class of embeddings can result in overfitting the source data and the matching of source and target distributions, resulting in arbitrarily high target risk. Hence, a restriction is needed. We observe that indeed, without appropriately constraining the embedding complexity, the performance of state-of-the-art methods such as domain-adversarial neural networks (Ganin et al., 2016) can deteriorate significantly. Next, we tailor the bound to multilayer neural networks. In a realistic scenario, one may have a total depth budget and divide the network into an encoder (embedding) and predictor by aligning the representations of source and target in a chosen layer, which defines the division. In this case, a more complex encoder necessarily implies a weaker predictor, and vice versa. This tradeoff is reflected in the bound and, we see that, in practice, there is an "optimal" division. To better optimize the tradeoff between encoder and predictor without having to tune the division, we propose to optimize the tradeoffs in all layers jointly via a simple yet effective objective that can easily be combined with most current approaches for learning domain-invariant representations. Implicitly, this objective restricts the more powerful deeper encoders by encouraging a simultaneous alignment across layers. In practice, the resulting algorithm achieves performance on par with or better than standard domain-invariant representations, without tuning of the division. Empirically, we examine our theory and learning algorithms on sentiment analysis (Amazon review dataset), digit classification (MNIST, MNIST-M, SVHN) and general object classification (Office-31). In short, this work makes the following contributions: • General upper bounds on target error that capture the effect of embedding complexity when learning domain-invariant representations; • Fine-grained analysis for multilayer neural networks, and a new objective with implicit regularization that stabilizes and improves performance; • Empirical validation of the analyzed tradeoffs and proposed algorithm on several datasets. In this paper, we theoretically and empirically analyze the effect of embedding complexity on the target risk in domain-invariant representations. We find a complexity tradeoff that has mostly been overlooked by previous work. In fact, without carefully selecting and restricting the encoder class, learning domain invariant representations might even harm the performance. We further develop a simple yet effective algorithm to approximately optimize the tradeoff, achieving performance across tasks that matches the best network division, i.e., complexity tradeoff. Interesting future directions of work include other strategies for model selection, and a more refined analysis and exploitation of the effect of inductive bias.
We study the effect of the embedding complexity in learning domain-invariant representations and develop a strategy that mitigates sensitivity to it.
{"src": "scitldr-AIC_train", "paper_id": "SkgOzlrKvH"}