This paper presents a generic framework to tackle the crucial class mismatch problem in unsupervised domain adaptation (UDA) for multi-class distributions.   Previous adversarial learning methods condition domain alignment only on pseudo labels, but noisy and inaccurate pseudo labels may perturb the multi-class distribution embedded in probabilistic predictions, hence bringing insufficient alleviation to the latent mismatch problem.   Compared with pseudo labels, class prototypes are more accurate and reliable since they summarize over all the instances and are  able  to  represent  the  inherent  semantic  distribution  shared  across  domains. Therefore, we propose a novel Prototype-Assisted Adversarial Learning (PAAL) scheme, which incorporates instance probabilistic predictions and class prototypes together  to  provide  reliable  indicators  for  adversarial  domain  alignment.    With the PAAL scheme,  we align both the instance feature representations and class prototype  representations  to  alleviate  the  mismatch  among  semantically  different classes.    Also,  we exploit the class prototypes as proxy to minimize the within-class variance in the target domain to mitigate the mismatch among semantically similar classes.   With these novelties, we constitute a Prototype-Assisted Conditional Domain Adaptation (PACDA) framework which well tackles the class mismatch problem. We demonstrate the good performance and generalization ability of the PAAL scheme and also PACDA framework on two UDA tasks, i.e., object recognition (Office-Home,ImageCLEF-DA, andOffice) and synthetic-to-real semantic segmentation (GTA5→CityscapesandSynthia→Cityscapes). Unsupervised domain adaptation (UDA) aims to leverage the knowledge of a labeled data set (source domain) to help train a predictive model for a unlabeled data set (target domain). Deep UDA methods bring noticeable performance gain to many tasks (Long et al., 2015; Saito et al., 2017; Richter et al., 2016; Tsai et al., 2018; Lee et al., 2019; Vu et al., 2019a) by exploiting supervision from heterogeneous sources. Some methods exploit maximum mean discrepancy (MMD) (Gretton et al., 2008; Long et al., 2015) or other distribution statistics like central moments (Sun & Saenko, 2016; Zellinger et al., 2017; Koniusz et al., 2017) for domain adaptation. Recently, generative adversarial learning (Goodfellow et al., 2014) provides a promising alternative solution to UDA problem. Since the labels of the target instances are not given in UDA, adversarial learning scheme for adaptation (Ganin & Lempitsky, 2015) suffers from the cross-domain misalignment, where the target instances from a class A are potentially misaligned with source instances from another class B. Inspired by the pseudo-labeling strategy from semi-supervised learning, previous methods either used the pseudo labels in the target domain to perform joint distribution discrepancy minimization (Long et al., 2013; or developed conditional adversarial learning methods that involve one high-dimensional domain discriminator or multiple discriminators (Chen et al., 2017b; Pei et al., 2018) . Though effective, these conditional domain adversarial learning methods align different instances from different domains relying only on their own predictions. Simple probabilistic predictions or pseudo labels may not accurately represent the semantic information of input instances, misleading the alignment. A toy example is given in Fig. 1(a) . The pseudo label of the chosen instance x is inclined to be class 'square' while the ground truth label is class 'circle'. Only guided by the instance prediction, the 'circle' class in the target domain and the 'square' class in the source domain are easily confused, causing the misalignment in the adversarial domain adaptation. To remedy the misalignment, we propose to exploit the class prototypes for adversarial domain alignment, instead of using only the possibly inaccurate predictions. Prototypes are global feature representations of different classes and are relevant to the inherent semantic structures shared across (a) conditional adversarial learning (b) prototype-assisted adversarial learning Figure 1: Illustration of two adversarial learning schemes. Different from class-agnostic adversarial learning that pursues the marginal distribution alignment but ignores the semantic consistency, (a) conditional adversarial learning relies heavily on the instance-level pseudo labels to perform conditional distribution alignment, while (b) our prototype-assisted adversarial learning integrates the instance-level pseudo labels and global class prototypes to make the conditional indicators more reliable. Class information is denoted in different shapes with source in solid and target in hollow. domains. As shown in Fig. 1(b) , class prototypes are expected to remedy the negative effects of inaccurate probabilistic predictions. Motivated by this, we propose a Prototype-Assisted Adversarial Learning (PAAL) scheme which complements instance predictions with class prototypes to obtain more reliable conditional information for guiding the source-target feature representation alignment. Specifically, we summarize the class prototypes from all instances according to their predictions. In this way, on one hand, we lower the dependence of class prototypes on instance predictions which may be inaccurate, and on the other hand, we encourage the instances with greater certainty to contribute more to their corresponding class prototypes. The prototypes are updated dynamically through a moving average strategy to make them more accurate and reliable. Then by broadcasting class prototypes to each instance according to its probability prediction, the inaccurate semantic distribution depicted by instance predictions can be alleviated. Based on reliable prototype-based conditional information, we align both the instance feature representations and the class prototypes through the proposed PAAL scheme to relieve the alignment among semantically dissimilar instances. However, such a conditional domain alignment may promote the confusion among semantically similar instances across domains to some degree. To further alleviate it, we introduce an intra-class objective in the target domain to pursue the class compactness. Built on the proposed PAAL scheme and this intra-class compactness objective, we develop a Prototype-Assisted Conditional Domain Adaptation (PACDA) framework for solving UDA problems. Extensive experimental evaluations on both object recognition and semantic segmentation tasks clearly demonstrate the advantages of our approaches over previous state-of-the-arts Xu et al., 2019; Tsai et al., 2019) . The contributions of this work can be summarized into three folds: 1) To the best of our knowledge, we are the first to leverage the class prototypes in conditional adversarial learning to prevent the misalignment in UDA; 2) We propose a simple yet effective domain adversarial learning framework PACDA to remedy the misalignment among semantically similar instances as well as semantically dissimilar instances; 3) The proposed PAAL scheme and PACDA framework are generic, and our framework achieves the state-of-the-art results on several unsupervised domain adaptation tasks including object recognition and semantic segmentation. In this work, we developed the prototype-assisted adversarial learning scheme to remedy the misalignment for UDA tasks. Unlike previous conditional ones whose performance is vulnerable to inaccurate instance predictions, our proposed scheme leverages the reliable and accurate class prototypes for aligning multi-class distributions across domains and is demonstrated to be more effective to prevent the misalignment. Then we further augment this scheme by imposing the intra-class compactness with the prototypes as proxy. Extensive evaluations on both object recognition and semantic segmentation tasks clearly justify the effectiveness and superiority of our UDA methods over well-established baselines.
We propose a reliable conditional adversarial learning scheme along with a simple, generic yet effective framework for UDA tasks.
{"src": "scitldr-AIC_train", "paper_id": "Byg79h4tvB"}