Deep transfer learning recently has acquired significant research interest. It makes use of pre-trained models that are learned from a source domain, and utilizes these models for the tasks in a target domain. Model-based deep transfer learning is probably the most frequently used method. However, very little research work has been devoted to enhancing deep transfer learning by focusing on the influence of data. In this paper, we propose an instance-based approach to improve deep transfer learning in a target domain. Specifically, we choose a pre-trained model from a source domain and apply this model to estimate the influence of training samples in a target domain. Then we optimize the training data of the target domain by removing the training samples that will lower the performance of the pre-trained model. We later either fine-tune the pre-trained model with the optimized training data in the target domain, or build a new model which is initialized partially based on the pre-trained model, and fine-tune it with the optimized training data in the target domain. Using this approach, transfer learning can help deep learning models to capture more useful features. Extensive experiments demonstrate the effectiveness of our approach on boosting the quality of deep learning models for some common computer vision tasks, such as image classification.
Although a lot of research works have been focusing on designing powerful network models to boost learning performance, very little progress has been made on investigating the influence of training data on learning effects. Koh and Liang in @cite_0 pioneer to study how a single training sample will influence the testing loss of a single testing sample. Inspired by their work, in @cite_23 propose a scheme to optimize training data, namely . In our work, we modify their scheme to optimize the training data in a target domain, aiming to implement our instance-based deep transfer learning approach. To ease the discussion, we use @math to denote this modified version, which is detailed in the . @math ( @math ) denotes a single training sample, and @math a model such as CNN with input @math . Let @math be the loss, and @math the influence of removing a training sample @math on the loss at a validation sample @math , where @math and @math equals the number of validation samples. If validation data is not given in advance, one can build a validation set by randomly separating 10
@cite_0 @cite_23
2949780682 2889565680
How can we explain the predictions of a black-box model? In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. Deep learning models learn to fit training data while they are highly expected to generalize well to testing data. Most works aim at finding such models by creatively designing architectures and fine-tuning parameters. To adapt to particular tasks, hand-crafted information such as image prior has also been incorporated into end-to-end learning. However, very little progress has been made on investigating how an individual training sample will influence the generalization ability of a model. In other words, to achieve high generalization accuracy, do we really need all the samples in a training dataset? In this paper, we demonstrate that deep learning models such as convolutional neural networks may not favor all training samples, and generalization accuracy can be further improved by dropping those unfavorable samples. Specifically, the influence of removing a training sample is quantifiable, and we propose a Two-Round Training approach, aiming to achieve higher generalization accuracy. We locate unfavorable samples after the first round of training, and then retrain the model from scratch with the reduced training dataset in the second round. Since our approach is essentially different from fine-tuning or further training, the computational cost should not be a concern. Our extensive experimental results indicate that, with identical settings, the proposed approach can boost performance of the well-known networks on both high-level computer vision problems such as image classification, and low-level vision problems such as image denoising.