An important question in task transfer learning is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task. Typically, transferability is either measured experimentally or inferred through task relatedness, which is often defined without a clear operational meaning. In this paper, we present a novel metric, H-score, an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems. Inspired by a principled information theoretic approach, H-score has a direct connection to the asymptotic error probability of the decision function based on the transferred feature. This formulation of transferability can further be used to select a suitable set of source tasks in task transfer learning problems or to devise efficient transfer learning policies. Experiments using both synthetic and real image data show that not only our formulation of transferability is meaningful in practice, but also it can generalize to inference problems beyond classification, such as recognition tasks for 3D indoor-scene understanding. Transfer learning is a learning paradigm that exploits relatedness between different learning tasks in order to gain certain benefits, e.g. reducing the demand for supervision BID22 ). In task transfer learning, we assume that the input domain of the different tasks are the same. Then for a target task T T , instead of learning a model from scratch, we can initialize the parameters from a previously trained model for some related source task T S . For example, deep convolutional neural networks trained for the ImageNet classification task have been used as the source network in transfer learning for target tasks with fewer labeled data BID7 ), such as medical image analysis BID24 ) and structural damage recognition in buildings (Gao & Mosalam) . An imperative question in task transfer learning is transferability, i.e. when a transfer may work and to what extent. Given a metric capable of efficiently and accurately measuring transferability across arbitrary tasks, the problem of task transfer learning, to a large extent, is simplified to search procedures over potential transfer sources and targets as quantified by the metric. Traditionally, transferability is measured purely empirically using model loss or accuracy on the validation set (Yosinski et al. (2014) ; Zamir et al. (2018) ; BID5 ). There have been theoretical studies that focus on task relatedness BID1 ; BID19 ; BID21 ; BID2 ). However, they either cannot be computed explicitly from data or do not directly explain task transfer performance. In this study, we aim to estimate transferability analytically, directly from the training data.We quantify the transferability of feature representations across tasks via an approach grounded in statistics and information theory. The key idea of our method is to show that the error probability of using a feature of the input data to solve a learning task can be characterized by a linear projection of this feature between the input and output domains. Hence we adopt the projection length as a metric of the feature's effectiveness for the given task, and refer to it as the H-score of the feature. More generally, H-score can be applied to evaluate the performance of features in different tasks, and is particularly useful to quantify feature transferability among tasks. Using this idea, we define task transferability as the normalized H-score of the optimal source feature with respect to the target task.As we demonstrate in this paper, the advantage of our transferability metric is threefold. (i) it has a strong operational meaning rooted in statistics and information theory; (ii) it can be computed directly and efficiently from the input data, with fewer samples than those needed for empirical learning; (iii) it can be shown to be strongly consistent with empirical transferability measurements.In this paper, we will first present the theoretical results of the proposed transferability metric in Section 2-4. Section 5 presents several experiments on real image data , including image classificaton tasks using the Cifar 100 dataset and 3D indoor scene understanding tasks using the Taskonomy dataset created by Zamir et al. (2018) . A brief review of the related works is included in Section 6. In this paper, we presented H-score, an information theoretic approach to estimating the performance of features when transferred across classification tasks. Then we used it to define a notion of task transferability in multi-task transfer learning problems, that is both time and sample complexity efficient. The resulting transferability metric also has a strong operational meaning as the ratio between the best achievable error exponent of the transferred representation and the minium error exponent of the target task.Our transferability score successfully predicted the performance for transfering features from ImageNet-1000 classification task to Cifar-100 task. Moreover, we showed how the transferability metric can be applied to a set of diverse computer vision tasks using the Taskonomy dataset.In future works, we plan to extend our theoretical results to non-classification tasks, as well as relaxing the local assumptions on the conditional distributions of the tasks. We will also investigate properties of higher order transferability, developing more scalable algorithms that avoid computing the H-score of all task pairs. On the application side, as transferability tells us how different tasks are related, we hope to use this information to design better task hierarchies for transfer learning. DISPLAYFORM0 x m with the following hypotheses: DISPLAYFORM1 Let P x m be the empirical distribution of the samples. The optimal test, i.e., the log likelihood ratio test can be stated in terms of information-theoretic quantities as follows: DISPLAYFORM2 Figure 10: The binary hypothesis testing problem. The blue curves shows the probility density functions for P 1 and P 2 . The rejection region and the acceptance region are highlighted in red and blue, respectively. The vertical line indicates the decision threshold.Further, using Sannov's theorem, we have that asymptotically the probability of type I error DISPLAYFORM3 where P * DISPLAYFORM4 m log T } denotes the rejection region. Similarly, for type II error DISPLAYFORM5 where P * 2 = argmin P ∈A D(P ||P 2 ) and A = {x m : FIG1 The overall probability of error is P (m) e = αP r(H 0 ) + βP r(H 1 ) and the best achievable exponent in the Bayesian probability of error (a.k.a. Chernoff exponent) is defined as: DISPLAYFORM6 DISPLAYFORM7 See Cover & BID6 for more background information on error exponents and its related theorems.Under review as a conference paper at ICLR 2019
We present a provable and easily-computable evaluation function that estimates the performance of transferred representations from one learning task to another in task transfer learning.
{"src": "scitldr-AIC_train", "paper_id": "BkxAUjRqY7"}