We study the problem of training machine learning models incrementally using active learning with access to imperfect or noisy oracles. We specifically consider the setting of batch active learning, in which multiple samples are selected as opposed to a single sample as in classical settings so as to reduce the training overhead. Our approach bridges between uniform randomness and score based importance sampling of clusters when selecting a batch of new samples. Experiments on
 benchmark image classification datasets (MNIST, SVHN, and CIFAR10) shows improvement over existing active learning strategies. We introduce an extra denoising layer to deep networks to make active learning robust to label noises and show significant improvements.
 Supervised learning is the most widely used machine learning method, but it requires labelled data for training. It is time-consuming and labor-intensive to annotate a large dataset for complex supervised machine learning models. For example, ImageNet (Russakovsky et al., 2015) reported the time taken to annotate one object to be roughly 55 seconds. Hence an active learning approach which selects the most relevant samples for annotation to incrementally train machine learning models is a very attractive avenue, especially for training deep networks for newer problems that have littel annotated data. Classical active learning appends the training dataset with a single sample-label pair at a time. Given the increasing complexity of machine learning models, it is natural to expand active learning procedures to append a batch of samples at each iteration instead of just one. Keeping such training overhead in mind, a few batch active learning procedures have been developed in the literature (Wei et al., 2015; Sener & Savarese, 2018; Sinha et al., 2019) . When initializing the model with a very small seed dataset, active learning suffers from the coldstart problem: at the very beginning of active learning procedures, the model is far from being accurate and hence the inferred output of the model is incorrect/uncertain. Since active learning relies on output of the current model to select next samples, a poor initial model leads to uncertain estimation of selection criteria and selection of wrong samples. Prior art on batch active learning suffers performance degradation due to this cold-start problem. Most active learning procedures assume the oracle to be perfect, i.e., it can always annotate samples correctly. However, in real-world scenarios and given the increasing usage of crowd sourcing, for example Amazon Mechanical Turk (AMT), for labelling data, most oracles are noisy. The noise induced by the oracle in many scenarios is resolute. Having multiple annotations on the same sample cannot guarantee noise-free labels due to the presence of systematic bias in the setup and leads to consistent mistakes. To validate this point, we ran a crowd annotation experiment on ESC50 dataset (Piczak, 2015) : each sample is annotated by 5 crowdworkers on AMT and the majority vote of the 5 annotations is considered the label. It turned out for some classes, 10% of the samples are annotated wrong, even with 5 annotators. Details of the experiment can be found in Appendix A. Under such noisy oracle scenarios, classical active learning algorithms such as (Chen et al., 2015a) under-perform as shown in Figure 1 . Motivating from these observations, we fashion a batch active learning strategy to be robust to noisy oracles. The main contributions of this work are as follows: (1) we propose a batch sample selection method based on importance sampling and clustering which caters to drawing a batch which is simultaneously diverse and important to the model; (2) we incorporate model uncertainty into the sampling probability to compensate poor estimation of the Noise channel is assumed to be a 10-symmetric channel, where ε is the probability of label error. importance scores when the training data is too small to build a meaningful model; (3) we introduce a denoising layer to deep networks to robustify active learning to noisy oracles. Main results, as shown in Fig. 3 demonstrate that in noise-free scenario, our method performs as the best over the whole active learning procedure, and in noisy scenario, our method outperforms significantly over state-of-the-art methods. In this paper we have proposed a batch sample selection mechanism for active learning with access to noisy oracles. We use mutual information between model parameters and the predicted class probabilities as importance score for each sample, and cluster the pool sample space with JensonShannon distance. We incorporate model uncertainty/confidence into Gibbs distribution over the clusters and select samples from each cluster with importance sampling. We introduce an additional layer at the output of deep networks to estimate label noise. Experiments on MNIST, SVHN, and CIFAR10 show that the proposed method is more robust against noisy labels compared with the state of the art. Even in noise-free scenarios, our method still performs the best for all three datasets. Our contributions open avenues for exploring applicability of batch active learning in setups involving imperfect data acquisition schemes either by construction or because of resource constraints. Under review as a conference paper at ICLR 2020 A ESC50 CROWD LABELING EXPERIMENT We selected 10 categories of ESC50 and use Amazon Mechanical Turk for annotation. In each annotation task, the crowd worker is asked to listen to the sound track and pick the class that the sound belongs to, with confidence level. The crowd worker can also pick "Unsure" if he/she does not think the sound track clearly belongs to one of the 10 categories. For quality control, we embed sound tracks that clearly belong to one class (these are called gold standards) into the set of tasks an annotator will do. If the annotator labels the gold standard sound tracks wrong, then labels from this annotator will be discarded. The confusion table of this crowd labeling experiment is shown in Figure 5 : each row corresponds to sound tracks with one ground truth class, and the columns are majority-voted crowd-sourced labels of the sound tracks. We can see that for some classes, such as frog and helicopter, even with 5 crowd workers, the majority vote of their annotation still cannot fully agree with the ground truth class.
We address the active learning in batch setting with noisy oracles and use model uncertainty to encode the decision quality of active learning algorithm during acquisition.
{"src": "scitldr-AIC_train", "paper_id": "SJxIkkSKwB"}