Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Models <s> Label noise is an important issue in classification, with many potential negative consequences. For example, the accuracy of predictions may decrease, whereas the complexity of inferred models and the number of necessary training samples may increase. Many works in the literature have been devoted to the study of label noise and the development of techniques to deal with label noise. However, the field lacks a comprehensive survey on the different types of label noise, their consequences and the algorithms that consider label noise. This paper proposes to fill this gap. First, the definitions and sources of label noise are considered and a taxonomy of the types of label noise is proposed. Second, the potential consequences of label noise are discussed. Third, label noise-robust, label noise cleansing, and label noise-tolerant algorithms are reviewed. For each category of approaches, a short discussion is proposed to help the practitioner to choose the most suitable technique in its own particular field of application. Eventually, the design of experiments is also discussed, what may interest the researchers who would like to test their own algorithms. In this paper, label noise consists of mislabeled instances: no additional information is assumed to be available like e.g., confidences on labels. <s> BIB001 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Models <s> Data are often labeled by many different experts with each expert only labeling a small fraction of the data and each data point being labeled by several experts. This reduces the workload on individual experts and also gives a better estimate of the unobserved ground truth. When experts disagree, the standard approaches are to treat the majority opinion as the correct label or to model the correct label as a distribution. These approaches, however, do not make any use of potentially valuable information about which expert produced which label. To make use of this extra information, we propose modeling the experts individually and then learning averaging weights for combining them, possibly in sample-specific ways. This allows us to give more weight to more reliable experts and take advantage of the unique strengths of individual experts at classifying certain types of data. Here we show that our approach leads to improvements in computer-aided diagnosis of diabetic retinopathy. We also show that our method performs better than competing algorithms by Welinder and Perona (2010), and by Mnih and Hinton (2012). Our work offers an innovative approach for dealing with the myriad real-world settings that use expert opinions to define labels for training. <s> BIB002 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Models <s> Supervised learning depends on annotated examples, which are taken to be the \emph{ground truth}. But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk. Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem). Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples. This raises two fundamental questions: (1) How can we best learn from noisy workers? (2) How should we allocate our labeling budget to maximize the performance of a classifier? We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data. The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality. Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality. We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality is above a threshold. Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits. <s> BIB003
A detailed taxonomy of label noise is provided in BIB001 . In this work we will follow the same taxonomy with a little abuse of notation. Label noise can be affected by three factors: data features, true label of data and labeler characteristic. According to dependence of these factors, label noise can be categorized in three sub classes. Random noise is totally random and does not depend on neither instance features nor its true class. With a given probability p e label is changed from its true class. Y-dependent noise is independent of X but depends on Y . That means data from a particular class is more likely to be mislabeled. For example, in hand written digit recognition task, "3" and "8" much more likely to be confused with each other rather then "3" and "5". XY-dependent noise depends on both X and Y . As in the y-dependent case, objects from particular class may be more likely to be mislabeled. Moreover, chance of mislabeling may change according to data features. If an instance has similar features to another instance from another class it is more likely to be mislabeled. What is not considered here is the case of multi-labeled data, in which each instance has multiple labels given by different annotators. In that scenario, works shows that modeling the characteristic of each labeler and using this information during training, significantly boosts the performance BIB002 . Characteristic of labeler can be independent from both X and Y . For example in a crowd-sourced dataset, some labelers can be total spammers who label with a random selection BIB003 . The behavior of each labeler can be explained by one of the groups above. For example, a spammer can be categorized as random noise while a better labeler can be modeled by y-dependent or xy-dependent noise. Therefore characteristic of labeler is not introduced as an extra ingredient in these definitions.