Domain Adaptation (DA) approaches [1]}, [2]}, [3]} are proposed to bridge the gap between the source and target domains. These methods tend to align the two domains in latent feature space so that a classifier trained on source data can also be applied to target samples. Despite the fact that great success has been made on the image level classification task [4]}, [5]}, [6]}, applying the latent space adaptation to semantic segmentation is non-trivial. The reasons are summarized as twofold. On the one hand, latent space adaptation for semantic segmentation may suffer from the complexity of high-dimensional features which encode various visual cues: appearance, shape and context, etc. Some of the task-independent nuisance factors might be easily involved in the encoded representation and mislead the domain alignment. On the other hand, in the adversarial domain adaptation framework [7]} which becomes popular in this field, the redundant information from the task-irrelevant factors might give excessive cues to the discriminator. The excessive cues lead the discriminator “unnecessary" high accuracy at the wrong time and produce uninformative gradients. All of this, unfortunately, will make the adversarial training process unstable, as pointed out in [8]}, [9]}, [10]}.
