Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> Label noise is an important issue in classification, with many potential negative consequences. For example, the accuracy of predictions may decrease, whereas the complexity of inferred models and the number of necessary training samples may increase. Many works in the literature have been devoted to the study of label noise and the development of techniques to deal with label noise. However, the field lacks a comprehensive survey on the different types of label noise, their consequences and the algorithms that consider label noise. This paper proposes to fill this gap. First, the definitions and sources of label noise are considered and a taxonomy of the types of label noise is proposed. Second, the potential consequences of label noise are discussed. Third, label noise-robust, label noise cleansing, and label noise-tolerant algorithms are reviewed. For each category of approaches, a short discussion is proposed to help the practitioner to choose the most suitable technique in its own particular field of application. Eventually, the design of experiments is also discussed, what may interest the researchers who would like to test their own algorithms. In this paper, label noise consists of mislabeled instances: no additional information is assumed to be available like e.g., confidences on labels. <s> BIB001 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> We present an approach to effectively use millions of images with noisy annotations in conjunction with a small subset of cleanly-annotated images to learn powerful image representations. One common approach to combine clean and noisy data is to first pre-train a network using the large noisy dataset and then fine-tune with the clean dataset. We show this approach does not fully leverage the information contained in the clean set. Thus, we demonstrate how to use the clean annotations to reduce the noise in the large dataset before fine-tuning the network using both the clean set and the full set with reduced noise. The approach comprises a multi-task network that jointly learns to clean noisy annotations and to accurately classify images. We evaluate our approach on the recently released Open Images dataset, containing ~9 million images, multiple annotations per image and over 6000 unique classes. For the small clean set of annotations we use a quarter of the validation set with ~40k images. Our results demonstrate that the proposed approach clearly outperforms direct fine-tuning across all major categories of classes in the Open Image dataset. Further, our approach is particularly effective for a large number of classes with wide range of noise in annotations (20-80% false positive annotations). <s> BIB002 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> Training deep neural networks requires many training samples, but in practice training labels are expensive to obtain and may be of varying quality, as some may be from trusted expert labelers while others might be from heuristics or other sources of weak supervision such as crowd-sourcing. This creates a fundamental quality versus-quantity trade-off in the learning process. Do we learn from the small amount of high-quality data or the potentially large amount of weakly-labeled data? We argue that if the learner could somehow know and take the label-quality into account when learning the data representation, we could get the best of both worlds. To this end, we propose"fidelity-weighted learning"(FWL), a semi-supervised student-teacher approach for training deep neural networks using weakly-labeled data. FWL modulates the parameter updates to a student network (trained on the task we care about) on a per-sample basis according to the posterior confidence of its label-quality estimated by a teacher (who has access to the high-quality labels). Both student and teacher are learned from the data. We evaluate FWL on two tasks in information retrieval and natural language processing where we outperform state-of-the-art alternative semi-supervised methods, indicating that our approach makes better use of strong and weak labels, and leads to better task-dependent data representations. <s> BIB003 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> Convolutional Neural Network (CNN) together with large-scale labeled data has achieved the state-of-theart accuracy in various computer vision tasks. In real-world settings, however, the labels of large scale data can be noisy, which shall seriously degenerate the performance of CNN. In this work, we propose a self-error-correcting CNN (SECCNN) to deal with the noisy labels problem, by simultaneously correcting the improbable labels and optimizing the deep model. Specifically, the SEC-CNN provides an opportunity to correct a wrong label by developing a confidence policy to switch between the label of the sample and the max-activated output neuron of the CNN. Based on the assumption that the deep model is more and more accurate during the training, the confidence policy relies more on the given labels at the beginning stages, but tends to believe that the max-activated neuron of the learned network is reliable. SEC-CNN enables CNN learning to be effective even with 80% noisy labels. Extensive experimental results on MNIST, CIFAR-10, ImageNet and CCFD face dataset demonstrate the effectiveness of the proposed method in dealing with noisy labels. <s> BIB004 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> Convolution neural network (CNN) has significantly pushed forward the development of face recognition and analysis techniques. Current CNN models tend to be deeper and larger to better fit large amounts of training data. When training data are from internet, their labels are often ambiguous and inaccurate. This paper presents a light CNN framework to learn a compact embedding on the large-scale face data with massive noisy labels. First, we introduce the concept of maxout activation into each convolutional layer of CNN, which results in a Max-Feature-Map (MFM). Different from Rectified Linear Unit that suppresses a neuron by a threshold (or bias), MFM suppresses a neuron by a competitive relationship. MFM can not only separate noisy signals and informative signals but also plays a role of feature selection. Second, a network of five convolution layers and four Network in Network (NIN) layers are implemented to reduce the number of parameters and improve performance. Lastly, a semantic bootstrapping method is accordingly designed to make the prediction of the models be better consistent with noisy labels. Experimental results show that the proposed framework can utilize large-scale noisy data to learn a light model in terms of both computational cost and storage space. The learnt single model with a 256-D representation achieves state-of-the-art results on five face benchmarks without fine-tuning. The light CNN model is released on this https URL <s> BIB005 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> To address the problem of incorrect labels in training data for deep learning, we propose a novel and simple training strategy, Iterative Cross Learning (ICL), that significantly improves the classification accuracy of neural networks with training data that has noisy labels. We randomly partition the noisy training data into multiple separate subsets, each of which is used to train an independent network. After these independent networks predict labels for the original data, if the labels agree, we update the label with the predicted result for that data point, but otherwise we update the label with a random label, a key to the success of our proposed method. The process is repeated, possibly with several stages, to gradually improve the performance. Testing our method on MNIST and CIFAR-10 with partially shuffled labels, ICL significantly improves the classification accuracy of existing methods when the data labels have noise, especially in heavy noise situations. Moreover, the proposed method doesn't require any change to the underlying neural networks' structure or loss function, so it can also easily be combined with other existing methods that address noisy labels, improving their performance. <s> BIB006 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> Deep neural networks (DNNs) trained on large-scale datasets have exhibited significant performance in image classification. Many large-scale datasets are collected from websites, however they tend to contain inaccurate labels that are termed as noisy labels. Training on such noisy labeled datasets causes performance degradation because DNNs easily overfit to noisy labels. To overcome this problem, we propose a joint optimization framework of learning DNN parameters and estimating true labels. Our framework can correct labels during training by alternating update of network parameters and labels. We conduct experiments on the noisy CIFAR-10 datasets and the Clothing1M dataset. The results indicate that our approach significantly outperforms other state-of-the-art methods. <s> BIB007 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> Crowdsourcing systems provide a cost effective and convenient way to collect labels, but they often fail to guarantee the quality of the labels. This paper proposes a novel framework that introduces noise correction techniques to further improve the quality of integrated labels that are inferred from the multiple noisy labels of objects. In the proposed general framework, information about the qualities of labelers estimated by a front-end ground truth inference algorithm is utilized to supervise subsequent label noise filtering and correction. The framework uses a novel algorithm termed adaptive voting noise correction (AVNC) to precisely identify and correct the potential noisy labels. After filtering out the instances with noisy labels, the remaining cleansed data set is used to create multiple weak classifiers, based on which a powerful ensemble classifier is induced to correct these noises. Experimental results on eight simulated data sets with different kinds of features and two real-world crowdsourcing data sets in different domains consistently show that: 1) the proposed framework can improve label quality regardless of inference algorithms, especially under the circumstance that each instance has a few repeated labels and 2) since the proposed AVNC algorithm considers both the number of and the probability of potential label noises, it outperforms the state-of-the-art noise correction algorithms. <s> BIB008 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> Label information plays an important role in a supervised hyperspectral image classification problem. However, current classification methods all ignore an important and inevitable problem—labels may be corrupted and collecting clean labels for training samples is difficult and often impractical. Therefore, how to learn from the database with noisy labels is a problem of great practical importance. In this paper, we study the influence of label noise on hyperspectral image classification and develop a random label propagation algorithm (RLPA) to cleanse the label noise. The key idea of RLPA is to exploit knowledge (e.g., the superpixel-based spectral–spatial constraints) from the observed hyperspectral images and apply it to the process of label propagation. Specifically, the RLPA first constructs a spectral–spatial probability transform matrix (SSPTM) that simultaneously considers the spectral similarity and superpixel-based spatial information. It then randomly chooses some training samples as “clean” samples and sets the rest as unlabeled samples, and propagates the label information from the “clean” samples to the rest unlabeled samples with the SSPTM. By repeating the random assignment (of “clean” labeled samples and unlabeled samples) and propagation, we can obtain multiple labels for each training sample. Therefore, the final propagated label can be calculated by a majority vote algorithm. Experimental studies show that the RLPA can reduce the level of noisy label and demonstrates the advantages of our proposed method over four major classifiers with a significant margin—the gains in terms of the average overall accuracy, average accuracy, and kappa are impressive, e.g., 9.18%, 9.58%, and 0.1043. The MATLAB source code is available at https://github.com/junjun-jiang/RLPA . <s> BIB009 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> Grading breast density is highly sensitive to normalization settings of digital mammogram as the density is tightly correlated with the distribution of pixel intensity. Also, the grade varies with readers due to uncertain grading criteria. These issues are inherent in the density assessment of digital mammography. They are problematic when designing a computer-aided prediction model for breast density and become worse if the data comes from multiple sites. In this paper, we proposed two novel deep learning techniques for breast density prediction: 1) photometric transformation which adaptively normalizes the input mammograms, and 2) label distillation which adjusts the label by using its output prediction. The photometric transformer network predicts optimal parameters for photometric transformation on the fly, learned jointly with the main prediction network. The label distillation, a type of pseudo-label techniques, is intended to mitigate the grading variation. We experimentally showed that the proposed methods are beneficial in terms of breast density prediction, resulting in significant performance improvement compared to various previous approaches. <s> BIB010 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> Deep learning has achieved excellent performance in various computer vision tasks, but requires a lot of training examples with clean labels. It is easy to collect a dataset with noisy labels, but such noise makes networks overfit seriously and accuracies drop dramatically. To address this problem, we propose an end-to-end framework called PENCIL, which can update both network parameters and label estimations as label distributions. PENCIL is independent of the backbone network structure and does not need an auxiliary clean dataset or prior information about noise, thus it is more general and robust than existing methods and is easy to apply. PENCIL outperformed previous state-of-the-art methods by large margins on both synthetic and real-world datasets with different noise types and noise rates. Experiments show that PENCIL is robust on clean datasets, too. <s> BIB011 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> ConvNets achieve good results when training from clean data, but learning from noisy labels significantly degrades performances and remains challenging. Unlike previous works constrained by many conditions, making them infeasible to real noisy cases, this work presents a novel deep self-learning framework to train a robust network on the real noisy datasets without extra supervision. The proposed approach has several appealing benefits. (1) Different from most existing work, it does not rely on any assumption on the distribution of the noisy labels, making it robust to real noises. (2) It does not need extra clean supervision or accessorial network to help training. (3) A self-learning framework is proposed to train the network in an iterative end-to-end manner, which is effective and efficient. Extensive experiments in challenging benchmarks such as Clothing1M and Food101-N show that our approach outperforms its counterparts in all empirical settings. <s> BIB012 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> B. Label Noise Cleansing <s> Video anomaly detection under weak labels is formulated as a typical multiple-instance learning problem in previous works. In this paper, we provide a new perspective, i.e., a supervised learning task under noisy labels. In such a viewpoint, as long as cleaning away label noise, we can directly apply fully supervised action classifiers to weakly supervised anomaly detection, and take maximum advantage of these well-developed classifiers. For this purpose, we devise a graph convolutional network to correct noisy labels. Based upon feature similarity and temporal consistency, our network propagates supervisory signals from high-confidence snippets to low-confidence ones. In this manner, the network is capable of providing cleaned supervision for action classifiers. During the test phase, we only need to obtain snippet-wise predictions from the action classifier without any extra post-processing. Extensive experiments on 3 datasets at different scales with 2 types of action classifiers demonstrate the efficacy of our method. Remarkably, we obtain the frame-level AUC score of 82.12% on UCF-Crime. <s> BIB013
A simple methodology to deal with noisy labels is to remove samples with suspicious labels or correct their noisy label to corresponding true class. This can be done in preprocessing stage of the training data, however such methods usually tackle the difficulty of distinguishing informative hard samples from those with noisy labels BIB001 . With superior classification abilities of deep neural networks, different methodologies to train label cleaning networks are proposed in literature. Unlike preprocessing of data, label cleaning models evaluated in this section are trained mutually with base classifier and their abilities keep improving during training. Methods under this category can be subdivided into two according to their need of reference set, label noise free subset of data, or not. 1) Using Reference Set: If there exists a clean subset of data, which has both noisy label and verified true label, network trained on this dataset can be used to relabel noisy labels until convergence BIB010 . In BIB005 , network trained on a clean dataset is fine tuned on noisy dataset for relabeling. Afterwards, network is again trained on relabeled data to re-sample dataset. In BIB002 label cleaning network gets two inputs: extracted features of instances by the base classifier and corresponding noisy label. It is trained to predict true class depending on these inputs and therefore learn the noise structure. For noisy data, base classifier is supervised by the predictions of label cleaning network. BIB003 uses teacher to clean labels for student to fine tune its training. First, student is trained on noisy data. Then features are extracted from clean data via the student model and teacher learns the structure of noise depending on these extracted features. Afterwards, teacher predicts soft labels for noisy data and student is again trained on these soft labels for fine tuning. Ensemble of networks which are trained with different subset of dataset are used in BIB006 . If they all agree on the label it is changed to predicted label, otherwise label is set to a random label. Authors argue that setting it to random would make noise more uniformly random and less structured which results in less bias in the learned network. 2) Not Using Reference Set: Joint optimization framework for both training classifier and propagating noisy labels to cleaner labels is presented in BIB007 . Their work consists of two progressive steps 1) Fix labels and update classifier weights with stochastic gradient descent (SGD), 2) Fix network parameters and update labels as moving average of network predictions. Additional regularization terms are added to cost label cleaning network for preventing it attaining one class posterior to all classes and base network predicting it. The same framework is used in BIB011 with probabilistic end-toend fashion, so that label posteriors are approximated with the help of classification and compatibility loss instead of moving average of network predictions. BIB008 proposes a two level approach. In the first stage, with any chosen inference algorithm, ground truth of the labels are determined and data is divided into two subset as noisy and clean. In the second stage, ensemble of weak classifiers are trained on clean data to predict true labels of noisy data. After that, these two subsets of data are merged to create the final enhanced dataset. BIB012 constructs prototypes, that are able to represent deep feature distribution of the corresponding class, for each class. Then corrected label is found by checking similarity among the data sample and prototypes. Using corrected label and noisy label, the final classifier network is trained. Assuming that correctly labeled data account for majority, BIB009 proposes to randomly split dataset to labeled and unlabeled subgroups. Then, labels are propagated to unlabeled data using similarity index among instances. This procedure repeated to produce multiple labels per instance and then final label is set with majority voting. BIB004 deploys a confidence policy where labels are determined by either network output or given noisy labels. With increasing number of epochs, more confidence is given to the network output defending that it gets better at predicting labels during training. BIB013 formulates video anomaly detection as a classification with label noise problem and trains a graph convolutional label noise cleaning network depending on features and temporal consistency of video snippets.