A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> FUNDAMENTALS OF BAYESIAN INFERENCE Probability and Inference Single-Parameter Models Introduction to Multiparameter Models Asymptotics and Connections to Non-Bayesian Approaches Hierarchical Models FUNDAMENTALS OF BAYESIAN DATA ANALYSIS Model Checking Evaluating, Comparing, and Expanding Models Modeling Accounting for Data Collection Decision Analysis ADVANCED COMPUTATION Introduction to Bayesian Computation Basics of Markov Chain Simulation Computationally Efficient Markov Chain Simulation Modal and Distributional Approximations REGRESSION MODELS Introduction to Regression Models Hierarchical Linear Models Generalized Linear Models Models for Robust Inference Models for Missing Data NONLINEAR AND NONPARAMETRIC MODELS Parametric Nonlinear Models Basic Function Models Gaussian Process Models Finite Mixture Models Dirichlet Process Models APPENDICES A: Standard Probability Distributions B: Outline of Proofs of Asymptotic Theorems C: Computation in R and Stan Bibliographic Notes and Exercises appear at the end of each chapter. <s> BIB001 </s> A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> Many applications of supervised learning require good generalization from limited labeled data. In the Bayesian setting, we can try to achieve this goal by using an informative prior over the parameters, one that encodes useful domain knowledge. Focusing on logistic regression, we present an algorithm for automatically constructing a multivariate Gaussian prior with a full covariance matrix for a given supervised learning task. This prior relaxes a commonly used but overly simplistic independence assumption, and allows parameters to be dependent. The algorithm uses other "similar" learning problems to estimate the covariance of pairs of individual parameters. We then use a semidefinite program to combine these estimates and learn a good prior for the current learning task. We apply our methods to binary text classification, and demonstrate a 20 to 40% test error reduction over a commonly used prior. <s> BIB002 </s> A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> An adapter with two or more tails for interconnecting a node to a multi-stage interconnection network. The plurality of tails create various options. Option 1 connects a 2-tailed adapter to two sub-networks comprising the same network to provide cost effective non-blocking networks. Both serial and parallel searches through the sub-networks for non-blocked paths are supported. Option 2 connects the 2-tailed adapter to redundant networks to provide fault tolerance in case of the failure of one network. Option 3 forms direct connections amongst two or three nodes without requiring a central switch. Instead, each tail provides a direct link to a different node. Option 4 extends the adapter to provide additional tails; 3-tailed adapters provide direct links amongst four nodes, 4-tailed adapters provide direct links amongst five nodes, etc. Option 5 interconnects similar multi-tailed adapters to provide ring or hyper-cube networks without requiring a central switch. <s> BIB003 </s> A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> A surgical ring clip (10) for ligating linear organ structures is provided by a solid ring (12) of a non-tissue reactive elastic material, having a central lumen (14). A plurality of pins (18) or teeth (28) are partially embedded within the elastic ring (12) with their exposed ends forming a pair of diametrically opposite files extending across and obstructing the lumen in an interleaved or meshed array when the ring is in its relaxed state. When released around a loop of a linear organ structure drawn through the lumen (14), the pins (18) or teeth (28) protrude through the loop, grasping any intermediate membrane and obstructing passage of the loop through the lumen (14), thereby preventing the ring clip (10) from slipping over the knuckle of the loop. <s> BIB004 </s> A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> This article focused on the definition and the study of a binary Bayesian criterion which measures a statistical agreement between a subjective prior and data information. The setting of this work is concrete Bayesian studies. It is an alternative and a complementary tool to the method recently proposed by Evans and Moshonov, [M. Evans and H. Moshonov, Checking for Prior-data conflict, Bayesian Anal. 1 (2006), pp. 893-914]. Both methods try to help the work of the Bayesian analyst, from preliminary to the posterior computation. Our criterion is defined as a ratio of Kullback-Leibler divergences; two of its main features are to make easy the check of a hierarchical prior and be used as a default calibration tool to obtain flat but proper priors in applications. Discrete and continuous distributions exemplify the approach and an industrial case study in reliability, involving the Weibull distribution, is highlighted. <s> BIB005 </s> A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> In transfer learning the aim is to solve new learning tasks using fewer examples by using information gained from solving related tasks. Existing transfer learning methods have been used successfully in practice and PAC analysis of these methods have been developed. But the key notion of relatedness between tasks has not yet been defined clearly, which makes it difficult to understand, let alone answer, questions that naturally arise in the context of transfer, such as, how much information to transfer, whether to transfer information, and how to transfer information across tasks. In this paper, we look at transfer learning from the perspective of Algorithmic Information Theory/Kolmogorov complexity theory, and formally solve these problems in the same sense Solomonoff Induction solves the problem of inductive inference. We define universal measures of relatedness between tasks, and use these measures to develop universally optimal Bayesian transfer learning methods. We also derive results in AIT that are interesting by themselves. To address a concern that arises from the theory, we also briefly look at the notion of Kolmogorov complexity of probability measures. Finally, we present a simple practical approximation to the theory to do transfer learning and show that even these are quite effective, allowing us to transfer across tasks that are superficially unrelated. The latter is an experimental feat which has not been achieved before, and thus shows the theory is also useful in constructing practical transfer algorithms. <s> BIB006 </s> A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> Multi-task learning is the problem of maximizing the performance of a system across a number of related tasks. When applied to multiple domains for the same task, it is similar to domain adaptation, but symmetric, rather than limited to improving performance on a target domain. We present a more principled, better performing model for this problem, based on the use of a hierarchical Bayesian prior. Each domain has its own domain-specific parameter for each feature but, rather than a constant prior over these parameters, the model instead links them via a hierarchical Bayesian global prior. This prior encourages the features to have similar weights across domains, unless there is good evidence to the contrary. We show that the method of (Daume III, 2007), which was presented as a simple "preprocessing step," is actually equivalent, except our representation explicitly separates hyperparameters which were tied in his work. We demonstrate that allowing different values for these hyperparameters significantly improves performance over both a strong baseline and (Daume III, 2007) within both a conditional random field sequence model for named entity recognition and a discriminatively trained dependency parser. <s> BIB007 </s> A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> Conventional human action recognition algorithms cannot work well when the amount of training videos is insufficient. We solve this problem by proposing a transfer topic model (TTM), which utilizes information extracted from videos in the auxiliary domain to assist recognition tasks in the target domain. The TTM is well characterized by two aspects: 1) it uses the bag-of-words model trained from the auxiliary domain to represent videos in the target domain; and 2) it assumes each human action is a mixture of a set of topics and uses the topics learned from the auxiliary domain to regularize the topic estimation in the target domain, wherein the regularization is the summation of Kullback-Leibler divergences between topic pairs of the two domains. The utilization of the auxiliary domain knowledge improves the generalization ability of the learned topic model. Experiments on Weizmann and KTH human action databases suggest the effectiveness of the proposed TTM for cross-domain human action recognition. <s> BIB008 </s> A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> High capacity classifiers, such as deep neural networks, often struggle on classes that have very few training examples. We propose a method for improving classification performance for such classes by discovering similar classes and transferring knowledge among them. Our method learns to organize the classes into a tree hierarchy. This tree structure imposes a prior over the classifier's parameters. We show that the performance of deep neural networks can be improved by applying these priors to the weights in the last layer. Our method combines the strength of discriminatively trained deep neural networks, which typically require large amounts of training data, with tree-based priors, making deep neural networks work well on infrequent classes as well. We also propose an algorithm for learning the underlying tree structure. Starting from an initial pre-specified tree, this algorithm modifies the tree to make it more pertinent to the task being solved, for example, removing semantic relationships in favour of visual ones for an image classification task. Our method achieves state-of-the-art classification results on the CIFAR-100 image data set and the MIR Flickr image-text data set. <s> BIB009 </s> A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> Scripts representing common sense knowledge about stereotyped sequences of events have been shown to be a valuable resource for NLP applications. We present a hierarchical Bayesian model for unsupervised learning of script knowledge from crowdsourced descriptions of human activities. Events and constraints on event ordering are induced jointly in one unified framework. We use a statistical model over permutations which captures event ordering constraints in a more flexible way than previous approaches. In order to alleviate the sparsity problem caused by using relatively small datasets, we incorporate in our hierarchical model an informed prior on word distributions. The resulting model substantially outperforms a state-of-the-art method on the event ordering task. <s> BIB010 </s> A review of single-source unsupervised domain adaptation <s> Bayesian inference <s> In total variability modeling, variable length speech utterances are mapped to fixed low-dimensional i-vectors. Central to computing the total variability matrix and i-vector extraction, is the computation of the posterior distribution for a latent variable conditioned on an observed feature sequence of an utterance. In both cases the prior for the latent variable is assumed to be non-informative, since for homogeneous datasets there is no gain in generality in using an informative prior. This work shows in the heterogeneous case, that using informative priors for computing the posterior, can lead to favorable results. We focus on modeling the priors using minimum divergence criterion or factor analysis techniques. Tests on the NIST 2008 and 2010 Speaker Recognition Evaluation (SRE) dataset show that our proposed method beats four baselines: For i-vector extraction using an already trained matrix, for the short2-short3 task in SRE’08, five out of eight female and four out of eight male common conditions, were improved. For the core-extended task in SRE’10, four out of nine female and six out of nine male common conditions were improved. When incorporating prior information into the training of the ${\rm T}$ matrix itself, the proposed method beats the baselines for six out of eight female and five out of eight male common conditions, for SRE’08, and five and six out of nine conditions, for the male and female case, respectively, for SRE’10. Tests using factor analysis for estimating priors show that two priors do not offer much improvement, but in the case of three separate priors (sparse data), considerable improvements were gained. <s> BIB011
In Bayesian inference, one forms a data likelihood function given a set of parameters and poses a prior distribution over these parameters BIB001 . Once data has been observed, one can derive the posterior distribution of the parameters, and make decisions based on Maximum A Posteriori estimation. Crucial to this inference process is the choice of prior distribution. Often, the shape of this prior distribution depends on what the expert believes are suitable values. In the absence of an expert, so-called non-informative priors can at times be constructed. In domain adaptation problems, the labeled source domain acts as a form of prior knowledge. One way to incorporate this knowledge in the inference process, is by fitting the prior distribution to the source data. For example, in NLP classification tasks with bag-of-words feature vectors, we know that features correlate heavily with each other BIB002 . Suppose a prior distribution is set on parameters of a linear classifier where each feature corresponds to a word. A noninformative prior for the classifier's parameters might look like a diagonal covariance matrix, i.e. no word correlations. But if one has access to a large data set of text documents, e.g. Wikipedia, one could estimate correlations between words. These estimates can replace the covariance matrix of the prior distribution. It is now informative as it contains knowledge gained from the source domain, and can improve performance of the target classifier. Note that this is a form of empirical Bayes as the prior is estimated from data  . The main difference from standard Empirical Bayes is that the data used for fitting the prior originates from a different distribution than the data that will be used for the likelihood function. In a sense, the source domain informs the model on what kind of feature structure could be expected in the target domain BIB004 . If the domains differ too much, then the informative prior will concentrate on a region of parameter space that is further away from the posterior induced by the observed data, than the non-informative prior. Hence, it would produce more uncertain predictions. But one can always check the appropriateness of such a prior after inference has been done BIB003 , BIB005 . Constructing informative priors using the source domain has been done in a number of settings: the recognition of speech utterances of variable length BIB011 , script knowledge induction BIB010 , cross-domain action recognition BIB008 , using decision-tree priors BIB006 , BIB009 and using hierarchical priors BIB007 .