We present SOSELETO (SOurce SELEction for Target Optimization), a new method for exploiting a source dataset to solve a classification problem on a target dataset.   SOSELETO is based on the following simple intuition: some source examples are more informative than others for the target problem.   To capture this intuition, source samples are each given weights; these weights are solved for jointly with the source and target classification problems via a bilevel optimization scheme.   The target therefore gets to choose the source samples which are most informative for its own classification task.   Furthermore, the bilevel nature of the optimization acts as a kind of regularization on the target, mitigating overfitting.   SOSELETO may be applied to both classic transfer learning, as well as the problem of training on datasets with noisy labels; we show state of the art results on both of these problems. Deep learning has demonstrated remarkable successes in tasks where large training sets are available. Yet, its usefulness is still limited in many important problems that lack such data. A natural question is then how one may apply the techniques of deep learning within these relatively data-poor regimes. A standard approach that seems to work relatively well is transfer learning. Despite its success, we claim that this approach misses an essential insight: some source examples are more informative than others for the target classification problem. Unfortunately, we don't know a priori which source examples will be important. Thus, we propose to learn this source filtering as part of an end-to-end training process.The resulting algorithm is SOSELETO: SOurce SELEction for Target Optimization. Each training sample in the source dataset is given a weight, representing its importance. A shared source/target representation is then optimized by means of a bilevel optimization. In the interior level, the source minimizes its classification loss with respect to the representation and classification layer parameters, for fixed values of the sample weights. In the exterior level, the target minimizes its classification loss with respect to both the source sample weights and its own classification layer. The sample weights implicitly control the representation through the interior level. The target therefore gets to choose the source samples which are most informative for its own classification task. Furthermore, the bilevel nature of the optimization acts as a kind of regularization on the target, mitigating overfitting, as the target does not directly control the representation parameters. The entire processtraining of the shared representation, source and target classifiers, and source weights -happens simultaneously.Related Work The most common techniques for transfer learning are feature extraction e.g. and fine-tuning, e.g. BID8 . An older survey of transfer learning techniques may be found in BID20 . Domain adaptation BID23 involves knowledge transfer when the source and target classes are the same. Earlier techniques aligned the source and target via matching of feature space statistics BID3 BID15 ; subsequent work used adversarial methods to improve the domain adaptation performance BID6 Tzeng et al., 2015; . In this paper, we are more interested in transfer learning where the source and target classes are different. BID16 ; BID21 BID1 b) address domain adaptation that is closer to our setting. BID2 examines "partial transfer learning" in which there is partial overlap between source and target classes (often the target classes are a subset of the source). This setting is also dealt with in BID0 . Like SOSELETO, BID7 propose selecting a portion of the source dataset, however, the selection is done prior to training and is not end-to-end. In , an adversarial loss aligns the source and target representations in a few-shot setting.Instance reweighting is a well studied technique for domain adaptation, demonstrated e.g. in Covariate Shift methods BID24 BID25 BID26 . While in these works, the source and target label spaces are the same, we allow them to be different -even entirely non-overlapping. Crucially, we do not make assumptions on the similarity of the distributions nor do we explicitly optimize for it. The same distinction applies for the recent work of BID9 , and for the partial overlap assumption of Zhang et al. (2018) . In addition, these two works propose an unsupervised approach, whereas our proposed method is completely supervised.Classification with noisy labels is a longstanding problem in the machine learning literature, see the review paper BID5 . Within the realm of deep learning, it has been observed that with sufficiently large data, learning with label noise -without modification to the learning algorithms -actually leads to reasonably high accuracy BID10 BID28 BID22 BID4 . We consider the setting where the large noisy dataset is accompanied by a small clean dataset. BID27 introduced a noise layer into the CNN that adapts the output to align with the noisy label distribution. Xiao et al. (2015) proposed to predict simultaneously the clean label and the type of noise; consider the same setting, but with additional information in the form of a knowledge graph on labels. BID18 conditioned the gradient propagation on the agreement of two separate networks. BID14 BID7 combine ideas of learning with label noise with instance reweighting. We have presented SOSELETO, a technique for exploiting a source dataset to learn a target classification task. This exploitation takes the form of joint training through bilevel optimization, in which the source loss is weighted by sample, and is optimized with respect to the network parameters; while the target loss is optimized with respect to these weights and its own classifier. We have empirically shown the effectiveness of the algorithm on both learning with label noise, as well as transfer learning problems. An interesting direction for future research involves incorporating an additional domain alignment term. We note that SOSELETO is architecture-agnostic, and may be extended beyond classification tasks. DISPLAYFORM0 end while SOSELETO consists of alternating the interior and exterior descent operations, along with the descent equations for the source and target classifiers φ s and φ t . As usual, the whole operation is done on a mini-batch basis, rather than using the entire set; note that if processing is done in parallel, then source mini-batches are taken to be non-overlapping, so as to avoid conflicts in the weight updates. A summary of SOSELETO algorithm appears in 1. Note that the target derivatives ∂L t /∂θ and ∂L t /∂φ t are evaluated over a target mini-batch; we suppress this for clarity.In terms of time-complexity, we note that each iteration requires both a source batch and a target batch; assuming identical batch sizes, this means that SOSELETO requires about twice the time as the ordinary source classification problem. Regarding space-complexity, in addition to the ordinary network parameters we need to store the source weights α. Thus, the additional relative spacecomplexity required is the ratio of the source dataset size to the number of network parameters. This is obviously problem and architecture dependent; a typical number might be given by taking the source dataset to be Imagenet ILSVRC-2012 (size 1.2M) and the architecture to be ResNeXt-101 Xie et al. (2017) (size 44.3M parameters), yielding a relative space increase of about 3%.
Learning with limited training data by exploiting "helpful" instances from a rich data source.  
{"src": "scitldr-AIC_train", "paper_id": "r1lqA-OSvV"}