Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> Many of the classification algorithms developed in the machine learning literature, including the support vector machine and boosting, can be viewed as minimum contrast methods that minimize a convex surrogate of the 0–1 loss function. The convexity makes these algorithms computationally efficient. The use of a surrogate, however, has statistical consequences that must be balanced against the computational virtues of convexity. To study these issues, we provide a general quantitative relationship between the risk as assessed using the 0–1 loss and the risk as assessed using any nonnegative surrogate loss function. We show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function—that it satisfies a pointwise form of Fisher consistency for classification. The relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications. We also present a refined version of this result in the... <s> BIB001 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> In this paper, we explore noise-tolerant learning of classifiers. We formulate the problem as follows. We assume that there is an unobservable training set that is noise free. The actual training set given to the learning algorithm is obtained from this ideal data set by corrupting the class label of each example. The probability that the class label of an example is corrupted is a function of the feature vector of the example. This would account for most kinds of noisy data one encounters in practice. We say that a learning method is noise tolerant if the classifiers learnt with noise-free data and with noisy data, both have the same classification accuracy on the noise-free data. In this paper, we analyze the noise-tolerance properties of risk minimization (under different loss functions). We show that risk minimization under 0-1 loss function has impressive noise-tolerance properties and that under squared error loss is tolerant only to uniform noise; risk minimization under other loss functions is not noise tolerant. We conclude this paper with some discussion on the implications of these theoretical results. <s> BIB002 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> When training a system to label images, the amount of labeled training data tends to be a limiting factor. We consider the task of learning to label aerial images from existing maps. These provide abundant labels, but the labels are often incomplete and sometimes poorly registered. We propose two robust loss functions for dealing with these kinds of label noise and use the loss functions to train a deep neural network on two challenging aerial image datasets. The robust loss functions lead to big improvements in performance and our best system substantially outperforms the best published results on the task we consider. <s> BIB003 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> In this paper, we theoretically study the problem of binary classification in the presence of random classification noise—the learner, instead of seeing the true labels, sees labels that have independently been flipped with some small probability. Moreover, random label noise is class-conditional— the flip probability depends on the class. We provide two approaches to suitably modify any given surrogate loss function. First, we provide a simple unbiased estimator of any loss, and obtain performance bounds for empirical risk minimization in the presence of iid data with noisy labels. If the loss function satisfies a simple symmetry condition, we show that the method leads to an efficient algorithm for empirical minimization. Second, by leveraging a reduction of risk minimization under noisy labels to classification with weighted 0-1 loss, we suggest the use of a simple weighted surrogate loss, for which we are able to obtain strong empirical risk bounds. This approach has a very remarkable consequence — methods used in practice such as biased SVM and weighted logistic regression are provably noise-tolerant. On a synthetic non-separable dataset, our methods achieve over 88% accuracy even when 40% of the labels are corrupted, and are competitive with respect to recently proposed methods for dealing with label noise in several benchmark datasets. <s> BIB004 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> Label noise is an important issue in classification, with many potential negative consequences. For example, the accuracy of predictions may decrease, whereas the complexity of inferred models and the number of necessary training samples may increase. Many works in the literature have been devoted to the study of label noise and the development of techniques to deal with label noise. However, the field lacks a comprehensive survey on the different types of label noise, their consequences and the algorithms that consider label noise. This paper proposes to fill this gap. First, the definitions and sources of label noise are considered and a taxonomy of the types of label noise is proposed. Second, the potential consequences of label noise are discussed. Third, label noise-robust, label noise cleansing, and label noise-tolerant algorithms are reviewed. For each category of approaches, a short discussion is proposed to help the practitioner to choose the most suitable technique in its own particular field of application. Eventually, the design of experiments is also discussed, what may interest the researchers who would like to test their own algorithms. In this paper, label noise consists of mislabeled instances: no additional information is assumed to be available like e.g., confidences on labels. <s> BIB005 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> In many applications, the training data, from which one needs to learn a classifier, is corrupted with label noise. Many standard algorithms such as SVM perform poorly in presence of label noise. In this paper we investigate the robustness of risk minimization to label noise. We prove a sufficient condition on a loss function for the risk minimization under that loss to be tolerant to uniform label noise. We show that the $0-1$ loss, sigmoid loss, ramp loss and probit loss satisfy this condition though none of the standard convex loss functions satisfy it. We also prove that, by choosing a sufficiently large value of a parameter in the loss function, the sigmoid loss, ramp loss and probit loss can be made tolerant to non-uniform label noise also if we can assume the classes to be separable under noise-free data distribution. Through extensive empirical studies, we show that risk minimization under the $0-1$ loss, the sigmoid loss and the ramp loss has much better robustness to label noise when compared to the SVM algorithm. <s> BIB006 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> Convex potential minimisation is the de facto approach to binary classification. However, Long and Servedio [2010] proved that under symmetric label noise (SLN), minimisation of any convex potential over a linear function class can result in classification performance equivalent to random guessing. This ostensibly shows that convex losses are not SLN-robust. In this paper, we propose a convex, classification-calibrated loss and prove that it is SLN-robust. The loss avoids the Long and Servedio [2010] result by virtue of being negatively unbounded. The loss is a modification of the hinge loss, where one does not clamp at zero; hence, we call it the unhinged loss. We show that the optimal unhinged solution is equivalent to that of a strongly regularised SVM, and is the limiting solution for any convex potential; this implies that strong l2 regularisation makes most standard learners SLN-robust. Experiments confirm the SLN-robustness of the unhinged loss. <s> BIB007 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> We prove that the empirical risk of most well-known loss functions factors into a linear term aggregating all labels with a term that is label free, and can further be expressed by sums of the loss. This holds true even for non-smooth, non-convex losses and in any RKHS. The first term is a (kernel) mean operator --the focal quantity of this work-- which we characterize as the sufficient statistic for the labels. The result tightens known generalization bounds and sheds new light on their interpretation. ::: Factorization has a direct application on weakly supervised learning. In particular, we demonstrate that algorithms like SGD and proximal methods can be adapted with minimal effort to handle weak supervision, once the mean operator has been estimated. We apply this idea to learning with asymmetric noisy labels, connecting and extending prior work. Furthermore, we show that most losses enjoy a data-dependent (by the mean operator) form of noise robustness, in contrast with known negative results. <s> BIB008 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> The convergence of Stochastic Gradient Descent SGD using convex loss functions has been widely studied. However, vanilla SGD methods using convex losses cannot perform well with noisy labels, which adversely affect the update of the primal variable in SGD methods. Unfortunately, noisy labels are ubiquitous in real world applications such as crowdsourcing. To handle noisy labels, in this paper, we present a family of robust losses for SGD methods. By employing our robust losses, SGD methods successfully reduce negative effects caused by noisy labels on each update of the primal variable. We not only reveal the convergence rate of SGD methods using robust losses, but also provide the robustness analysis on two representative robust losses. Comprehensive experimental results on six real-world datasets show that SGD methods using robust losses are obviously more robust than other baseline methods in most situations with fast convergence. <s> BIB009 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and on algorithms for estimating true labels from noisy labels. An alternate approach would be to look for loss functions that are inherently noise-tolerant. For binary classification there exist theoretical results on loss functions that are robust to label noise. In this paper, we provide some sufficient conditions on a loss function so that risk minimization under that loss function would be inherently tolerant to label noise for multiclass classification problems. These results generalize the existing results on noise-tolerant loss functions for binary classification. We study some of the widely used loss functions in deep networks and show that the loss function based on mean absolute value of error is inherently robust to label noise. Thus standard back propagation is enough to learn the true classifier even under label noise. Through experiments, we illustrate the robustness of risk minimization with such loss functions for learning neural networks. <s> BIB010 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> Deep neural networks (DNNs) have achieved tremendous success in a variety of applications across many disciplines. Yet, their superior performance comes with the expensive cost of requiring correctly annotated large-scale datasets. Moreover, due to DNNs' rich capacity, errors in training labels can hamper performance. To combat this problem, mean absolute error (MAE) has recently been proposed as a noise-robust alternative to the commonly-used categorical cross entropy (CCE) loss. However, as we show in this paper, MAE can perform poorly with DNNs and challenging datasets. Here, we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm, while yielding good performance in a wide range of noisy label scenarios. We report results from experiments conducted with CIFAR-10, CIFAR-100 and FASHION-MNIST datasets and synthetically generated noisy labels. <s> BIB011 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> We target at a challenging problem of training Deep Neural Networks (DNNs) on abnormal training data, where a considerable proportion of observations and their labels are semantically unmatched, e.g., corrupted labels, or out-of-distribution training examples etc. In this work, we analyse and address this challenge from the perspective of example weighting built in empirical loss functions, i.e., gradient magnitude with respect to logits, an angle that is not thoroughly studied so far. Concretely, we make two main contributions: 1) We present new observations and insightful analysis about mean absolute error (MAE), which is theoretically proved to be noise-robust. First, we reveal its underfitting problem in practice when severe noise exists. Second, we conclude its noise-robustness is from emphasising on uncertain examples instead of treating training samples equally, as claimed in prior work. Third, its underfitting issue is interpreted from the differentiation degree viewpoint. 2) We propose an effective and simple solution to enhance MAE's fitting ability while preserving its noise-robustness. Without changing the overall weighting scheme, i.e., what examples get higher weights, we simple adjust MAE's weighting variance, i.e., how much difference over examples' weights. For simplicity, we name our solution, Improved MAE (IMAE). We prove the effectiveness of IMAE with extensive experiments: image classification under symmetric label noise and real world unknown noise, and video retrieval which contains different types of noise. Source code: \url{this https URL}. <s> BIB012 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> A. Robust Losses <s> Training accurate deep neural networks (DNNs) in the presence of noisy labels is an important and challenging task. Though a number of approaches have been proposed for learning with noisy labels, many open issues remain. In this paper, we show that DNN learning with Cross Entropy (CE) exhibits overfitting to noisy labels on some classes ("easy" classes), but more surprisingly, it also suffers from significant under learning on some other classes ("hard" classes). Intuitively, CE requires an extra term to facilitate learning of hard classes, and more importantly, this term should be noise tolerant, so as to avoid overfitting to noisy labels. Inspired by the symmetric KL-divergence, we propose the approach of \textbf{Symmetric cross entropy Learning} (SL), boosting CE symmetrically with a noise robust counterpart Reverse Cross Entropy (RCE). Our proposed SL approach simultaneously addresses both the under learning and overfitting problem of CE in the presence of noisy labels. We provide a theoretical analysis of SL and also empirically show, on a range of benchmark and real-world datasets, that SL outperforms state-of-the-art methods. We also show that SL can be easily incorporated into existing methods in order to further enhance their performance. <s> BIB013
A loss function is said to be noise robust if the classifier learned with noisy and noise-free data, both achieve the same classification accuracy BIB002 . Algorithms under this section, aims to design loss function in such a way that the noise would not decrease the performance. However, it is shown that noise can badly affect the performance even for the robust loss functions BIB005 . In BIB002 , it is shown that certain non-convex loss functions, such as 0-1 loss, has noise tolerance much more than commonly used convex losses. Extending this work BIB006 derives sufficient conditions for a loss function to be noise tolerant under the risk minimization for uniform and non-uniform noises. Their work shows that if a loss function is symmetric, meaning that sum of its components are equal to a constant and noise level is below a threshold, under uniform noise it is noise robust. In this content, they empirically show that none of the standard convex loss functions has noise robustness while 0-1 loss has, under given circumstances. However, 0-1 loss is non-convex and non-differentiable, therefore surrogate loss of 0-1 loss is proposed in BIB001 , which is still noise sensitive. Multi-class classification is considered in BIB010 and it is shown that mean absolute value of error (MAE) is inherently robust to label noise under same circumstances. IMAE, which is an improved version of MAE, is proposed in BIB012 , where gradients are scaled with an hyper-parameter to adjusts MAE's weighting variance. BIB011 argues that MAE provides much smaller learning rate than categorical cross entropy (CCE), therefore a new loss function is suggested which combines robustness of MAE and implicit weighting of CCE. With a tuning parameter, it can be made more similar to MAE or CCE. Symmetric cross entropy is used in BIB013 , arguing that symmetry provides noise robust boosting in learning hard samples. Given that mean operator is not significantly affected by the noise, work of BIB008 showed that class of linear odd losses (LOLs) are robust to the label noise at some degree, such that difference between the risks of minimizers with noisy data and clean data are smaller than a predefined threshold. BIB007 shows that classification-calibrated loss functions are asymptotically robust to symmetric label noise. SGD method in general is analyzed under label noise in BIB009 and shown to be more robust than its counterparts. Given that noise prior is known BIB004 provides two surrogate loss functions namely unbiased and weighted estimator of the loss function using the prior information about label noise. BIB003 considers asymmetric omission noise for binary classification case, where the task is to find road pixels from a satellite map image. Omission noise makes network less confident about its predictions, so they modified cross-entropy loss to penalize network less for making wrong but confident predictions since these labels are more likely to be noisy.