Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Adding Synthetic Label Noise <s> Supervised learning depends on annotated examples, which are taken to be the \emph{ground truth}. But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk. Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem). Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples. This raises two fundamental questions: (1) How can we best learn from noisy workers? (2) How should we allocate our labeling budget to maximize the performance of a classifier? We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data. The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality. Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality. We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality is above a threshold. Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits. <s> BIB001 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Adding Synthetic Label Noise <s> Deep neural networks (DNNs) trained on large-scale datasets have exhibited significant performance in image classification. Many large-scale datasets are collected from websites, however they tend to contain inaccurate labels that are termed as noisy labels. Training on such noisy labeled datasets causes performance degradation because DNNs easily overfit to noisy labels. To overcome this problem, we propose a joint optimization framework of learning DNN parameters and estimating true labels. Our framework can correct labels during training by alternating update of network parameters and labels. We conduct experiments on the noisy CIFAR-10 datasets and the Clothing1M dataset. The results indicate that our approach significantly outperforms other state-of-the-art methods. <s> BIB002
There are several widely used methods to add synthetic label noise to the given dataset. Y-dependent and XY-dependent noise is considered in this section, since random noise can easily be added by changing labels randomly. All mentioned methods are for the case of single labeler. If one requires to create synthetic noise for multiple annotators, any of the given methods can be used multiple times with different parameters to mimic different labelers. 1) Y-Dependent Noise: Y-dependent noise can be represented with confusion-matrix, where the entry p ij represents the probability of flipping label of instance from class i to class j. Then samples from each class can be flipped to other classes with the given probabilities. Since confusion matrix consists of probabilities, it should be satisfied that p ij > 0 for ∀i, j and j=n j=0 p ij for ∀i where n is the number of classes. Uniform Noise: Confusion matrix is defined where probability of choosing any other class, rather than true class, is equally distributed. A predefined probability of mislabeling p can be defined and samples from each class is changed from its true class to any other classes with equal probability. Random Noise: Confusion matrix is defined where probability of choosing any other class, rather than true class, is randomly distributed. A stochastic probability of mislabeling p ij can be defined for each mislabeling probability from class i to j. Paired Noise: Any instance from a class can only be flipped to one other class with probability p. In this type of noise, sufficient condition for learning is p < %50, because otherwise pairwise mislabeled data takes the majority. Class Dependent Noise: Flip probabilities from class i to j is assigned according to similarity of classes. For example, in handwritten digit recognition task p 17 , which is probability of mislabeling '1' as '7', can be higher while p 15 is lower. These probabilities can be assigned by an human annotator or by the prediction probabilities of pre-trained network. 2) XY-Dependent Noise: Artificially creating this type of noise is more complicated than previous ones, since one needs to consider the feature vectors of each sample. Most obvious method is to use human annotator to choose samples, but this is both expensive and not scalable. One particular approach is to use pre-trained network, which is either directly training on given data  ,  or dataset from similar but different domain BIB002 , to extract decision boundaries . Differently, in case of the presence of surrounding texts for image, user defined texts can be used to extract mostly probable noisy label BIB001 .