A review of single-source unsupervised domain adaptation <s> PAC-Bayes <s> We provide a PAC-Bayesian bound for the expected loss of convex combinations of classifiers under a wide class of loss functions (which includes the exponential loss and the logistic loss). Our numerical experiments with Adaboost indicate that the proposed upper bound, computed on the training set, behaves very similarly as the true loss estimated on the testing set. <s> BIB001 </s> A review of single-source unsupervised domain adaptation <s> PAC-Bayes <s> We provide a first PAC-Bayesian analysis for domain adaptation (DA) which arises when the learning and test distributions differ. It relies on a novel distribution pseudodistance based on a disagreement averaging. Using this measure, we derive a PAC-Bayesian DA bound for the stochastic Gibbs classifier. This bound has the advantage of being directly optimizable for any hypothesis space. We specialize it to linear classifiers, and design a learning algorithm which shows interesting results on a synthetic problem and on a popular sentiment annotation task. This opens the door to tackling DA tasks by making use of all the PAC-Bayesian tools. <s> BIB002 </s> A review of single-source unsupervised domain adaptation <s> PAC-Bayes <s> We study the issue of PAC-Bayesian domain adaptation: We want to learn, from a source domain, a majority vote model dedicated to a target one. Our theoretical contribution brings a new perspective by deriving an upper-bound on the target risk where the distributions' divergence-- expressed as a ratio--controls the trade-off between a source error measure and the target voters' disagreement. Our bound suggests that one has to focus on regions where the source data is informative. From this result, we derive a PAC-Bayesian generalization bound, and specialize it to linear classifiers. Then, we infer a learning algorithm and perform experiments on real data. <s> BIB003
The PAC-Bayesian framework is a combination of PAClearning and Bayesian inference BIB002 ,  , BIB001 . One defines a prior distribution π over an hypothesis space H of classification functions BIB002 , BIB001 . Through observing data, one aims to obtain a posterior distribution ρ over H. The expected prediction with respect to hypotheses drawn from ρ is known as the ρ-weighted majority vote, or Bayes classifier  , BIB001 . So far, the framework resembles standard Bayesian inference. It departs from that by replacing the consideration of ρ over the entire hypothesis space by, what is known as, the Gibbs classifier BIB002 . The Gibbs classifier G ρ draws a single hypothesis from H according to ρ to make a prediction. In expectation, over hypotheses drawn from H, the Gibbs classifier is equivalent to the Bayes classifier. Using G ρ , a domain-adaptive PAC-Bayesian generalization error bound can be derived BIB002 . This bound is of similar form to the one in Section 2.4 consisting of the source risk of the Gibbs classifier plus a discrepancy term and the error of the ideal-joint-hypothesis. In this case, the discrepancy term consists of the difference between the disagreement of two hypotheses drawn from ρ on the source data versus the disagreement of two hypotheses drawn from ρ on the target data BIB002 : Building on the previous results, a new PAC-Bayesian bound has been derived that is not dependent on the error of the ideal joint hypothesis, Instead, it utilizes a term describing the part of the target domain for which the source domain is uninformative BIB003 : where T \ S denotes the part of feature space in the support of the target distribution but not in the support of the source distribution, and Supp refers to the support of a distribution. The supremum over the risk function denotes that this area is assigned the largest risk obtainable. This risk is then weighted by the probability under the target distribution. η T \S grows as the part of the target domain with no support from the source domain grows. The rest of the bound is formed by a re-defined disagreement term that is now specific to the target distribution, , and a factor weighting the source error BIB003 : This factor is an exponentiated Rényi divergence For α = 2, it would equal the Rényi divergence from the bound for importance weighting (c.f. Equation 5), except that it is with respect to the joint distribution in each domain instead of the data distributions. This divergence focuses on parts of feature space where the source domain's support is within the target domain's support. Effectively, the source error in parts of feature space supported by the source domain but not the target domain, has little to no influence. The above bound can be expressed in computable terms BIB003 . In that case, The Gibbs classifier becomes linear with ρ a distribution over classsifier parameters. The expected disagreement is bounded by the empirical disagreement and the Kullback-Leibler divergence between the prior defined over the hypothesis space and the posterior, normalized by the target sample size. For spherical Gaussian distributions over linear classifier parameters, this KL-divergence between prior and posterior constitutes the squared norm of the parameters. Similarly, the expected source error can be upper bounded by the empirical error and the Kullback-Leibler divergence between π and ρ normalized by the source sample size. The resulting optimization objective, called Domain Adaptation for Linear Classifiers (DALC), trades off the disagreement of the classifier on the target samples, the classification error on the source samples and the squared norm of the classifier parameters: where Φ is a probit loss function andΦ(x) = 2Φ(−x)Φ(x). µ and ζ are trade-off parameters based on the normalization factors for the empirical expected disagreement and the empirical source error, respectively. DALC resembles a standard linear classifier in that minimizes the average loss, here in the form of the squared probit function. However, it includes an additional term that penalizes predictions on the target samples that are far removed from 0.