In this work, we present a novel upper bound of target error to address the problem for unsupervised domain adaptation. Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks. Furthermore,  Ben-David et al. (2010) provide an upper bound for target error when transferring the knowledge, which can be summarized as minimizing the source error and  distance between marginal distributions simultaneously. However, common methods based on the theory usually ignore the joint error such that samples from different classes might be mixed together when matching marginal distribution. And in such case, no matter how we minimize the marginal discrepancy, the target error is not bounded due to an increasing joint error. To address this problem, we propose a general upper bound taking joint error into account, such that the undesirable case can be properly penalized. In addition, we utilize constrained hypothesis space to further formalize a tighter bound as well as a novel cross margin discrepancy to measure the dissimilarity between hypotheses which alleviates instability during adversarial learning. Extensive empirical evidence shows that our proposal outperforms related approaches in image classification error rates on standard domain adaptation benchmarks. The advent of deep convolutional neural networks (Krizhevsky et al., 2012) brings visual learning into a new era. However, the performance heavily relies on the abundance of data annotated with ground-truth labels. Since traditional machine learning assumes a model is trained and verified in a fixed distribution (single domain), where generalization performance is guaranteed by VC theory (N. Vapnik, 2000) , thus it cannot always be applied to real-world problem directly. Take image classification task as an example, a number of factors, such as the change of light, noise, angle in which the image is pictured, and different types of sensors, can lead to a domain-shift thus harm the performance when predicting on test data. Therefore, in many practical cases, we wish that a model trained in one or more source domains is also applicable to another domain. As a solution, domain adaptation (DA) aims to transfer the knowledge learned from a source distribution, which is typically fully labeled into a different (but related) target distribution. This work focus on the most challenging case, i.e, unsupervised domain adaptation (UDA), where no target label is available. Ben-David et al. (2010) suggests that target error can be minimized by bounding the error of a model on the source data, the discrepancy between distributions of the two domains, and a small optimal joint error. Owing to the strong representation power of deep neural nets, many researchers focus on learning domain-invariant features such that the discrepancy of two feature spaces can be minimized. For aligning feature distributions across domains, mainly two strategies have been substantially explored. The first one is bridging the distributions by matching all their statistics (Long et al., 2015; Pan et al., 2009) . The second strategy is using adversarial learning (Goodfellow et al., 2014) to build a minimax game between domain discriminator and feature extractor, where a domain discriminator is trained to distinguish the source from the target while the feature extractor is learned to confuse it simultaneously (Ganin & Lempitsky, 2015; Ganin et al., 2016; Tzeng et al., 2017) . In spite of the remarkable empirical results accomplished by feature distribution matching schemes, they still suffer from a major limitation: the joint distributions of feature spaces and categories are not well aligned across data domains. As is reported in Ganin et al. (2016) , such methods fail to generalize in certain closely related source/target pairs, e.g., digit classification adaptation from MNIST to SVHN. One potential reason is when matching marginal distributions of source and target domains, samples from different classes can be mixed together, where the joint error becomes nonnegligible since no hypothesis can classify source and target at the same time. This work aims to address the above problem by incorporating joint error to formalize an optimizable upper bound such that the undesired overlap due to a wrong match can be properly penalized. We evaluate our proposal on several different classification tasks. In some experimental settings, our method outperforms other methods by a large margin. The contributions of this work can be summarized as follows: · We propose a novel upper bound taking joint error into account and theoretically prove that our proposal can reduce to several other methods under certain simplifications. · We construct a constrained hypothesis space such that a much tighter bound can be obtained during optimization. · We adopt a novel measurement, namely cross margin discrepancy, for the dissimilarity of two hypotheses on certain domain to alleviate the instability during adversarial learning and provide reliable performance. In this work, we propose a general upper bound that takes the joint error into account. Then we further pursuit a tighter bound with reasonable constraint on the hypothesis space. Additionally, we adopt a novel cross domain discrepancy for dissimilarity measurement which alleviates the instability during adversarial learning. Extensive empirical evidence shows that learning an invariant representation is not enough to guarantee a good generalization in the target domain, as the joint error matters especially when the domain shift is huge. We believe our results take an important step towards understanding unsupervised domain adaptation, and also stimulate future work on the design of stronger adaptation algorithms that manage to align conditional distributions without using pseudo-labels from the target domain. layer and a 0.5 rate of dropout is conducted. Nesterov accelerated gradient is used for optimization with a mini-batch size of 32 and an initial learning rate of 10 −3 which decays exponentially. As for the hyper-parameter, we test for γ = {0.1, 0.5, 0.9, 1} and η = {0, 0.5, 0.8, 0.9}. For a direct comparison, we report the accuracy after 10 epochs.
joint error matters for unsupervised domain adaptation especially when the domain shift is huge
{"src": "scitldr-AIC_train", "paper_id": "rkerLaVtDr"}