Domain similarity measures can be used to gauge adaptability and select suitable data for transfer learning, but existing approaches define ad hoc measures that are deemed suitable for respective tasks. Inspired by work on curriculum learning, we propose to data selection measures using Bayesian Optimization and evaluate them across models, domains and tasks. Our learned measures outperform existing domain similarity measures significantly on three tasks: sentiment analysis, part-of-speech tagging, and parsing. We show the importance of complementing similarity with diversity, and that learned measures are -- to some degree -- transferable across models, domains, and even tasks.
There is a long history of research in adaptive data selection, with early approaches grounded in information theory using a Bayesian learning framework @cite_38 . It has also been studied extensively as active learning @cite_33 . Curriculum learning is related to active learning @cite_24 , whose view is different: active learning aims at finding the most difficult instances to label, examples typically close to the decision boundary. Confidence-based measures are prominent, but as such are less widely applicable than our model-agnostic approach.
@cite_24 @cite_38 @cite_33
2903158431 2115305054 190546068
Learning can be made more efficient if we can actively select particularly salient data points. Within a Bayesian learning framework, objective functions are discussed that measure the expected informativeness of candidate measurements. Three alternative specifications of what we want to gain information about lead to three different criteria for data selection. All these criteria depend on the assumption that the hypothesis space is correct, which may prove to be their main weakness. This chapter introduces the notions of rational expectations and optimal learning extensively used in economic theory.* It has become well known from recent literature that in active learning situations (where the actions of the statistician, or the person learning about some parameters, influences the draws from the distribution about which he she is learning), full learning may not take place. This challenges the use of the rational expectations hypothesis which is justified on the basis that agents operating in an economy eventually all learn the true structure of the economy and optimize accordingly. In El-Gamal and Sundaram (1989, 1990) we presented a framework where a Bayesian economist imposes priors on agent-priors and we then study the evolution of those economist beliefs. We showed that generically, the economist limit beliefs generically do not have point mass at any particular agent-belief, let alone the true rational expectations belief. We show, however, that in most cases where there is sufficient variability in the law of motion that the agents are trying to learn, in sequential models that are extensively used in the economic literature, the rational expectations hypothesis may indeed be justified on the basis of optimizing and optimally updating agents.