Recently deep neural networks have shown their capacity to memorize training data, even with noisy labels, which hurts generalization performance. To mitigate this issue, we propose a simple but effective method that is robust to noisy labels, even with severe noise.   Our objective involves a variance regularization term that implicitly penalizes the Jacobian norm of the neural network on the whole training set (including the noisy-labeled data), which encourages generalization and prevents overfitting to the corrupted labels. Experiments on noisy benchmarks demonstrate that our approach achieves state-of-the-art performance with a high tolerance to severe noise. Recently deep neural networks (DNNs) have achieved remarkable performance on many tasks, such as speech recognition Amodei et al. (2016) , image classification He et al. (2016) , object detection Ren et al. (2015) . However, DNNs usually need a large-scale training dataset to generalize well. Such large-scale datasets can be collected by crowd-sourcing, web crawling and machine generation with a relative low price, but the labeling may contain errors. Recent studies Zhang et al. (2016) ; Arpit et al. (2017) reveal that mislabeled examples hurt generalization. Even worse, DNNs can memorize the training data with completely randomly-flipped labels, which indicates that DNNs are prone to overfit noisy training data. Therefore, it is crucial to develop algorithms robust to various amounts of label noise that still obtain good generalization.To address the degraded generalization of training with noisy labels, one direct approach is to reweigh training examples Ren et al. (2018); Jiang et al. (2017) ; Han et al. (2018) ; Ma et al. (2018) , which is related to curriculum learning. The general idea is to assign important weights to examples with a high chance of being correct. However, there are two major limitations of existing methods. First, imagine an ideal weighting mechanism. It will only focus on the selected clean examples. For those incorrectly labeled data samples, the weights should be near zero. If a dataset is under 80% noise corruption, an ideal weighting mechanism assigns nonzero weights to only 20% examples and abandons the information in a large amount of 80% examples. This leads to an insufficient usage of training data. Second, previous methods usually need some prior knowledge on the noise ratio or the availability of an additional clean unbiased validation dataset. But it is usually impractical to get this extra information in real applications. Another approach is correction-based, estimating the noisy corruption matrix and correcting the labels Patrini et al. (2017) ; Reed et al. (2014) ; Goldberger & Ben-Reuven (2017) . But it is often difficult to estimate the underlying noise corruption matrix when the number of classes is large. Further, there may not be an underlying ground truth corruption process but an open set of noisy labels in the real world. Although many complex approaches Jiang et al. (2017); Ren et al. (2018); Han et al. (2018) have been proposed to deal with label noise, we find that a simple yet effective baseline can achieve surprisingly good performance compared to the strong competing methods.In this paper, we first analyze the conditions for good generalization. A model with simpler hypothesis and smoother decision boundaries can generalize better. Then we propose a new algorithm which can satisfy the conditions and take advantage of the whole dataset including the noisy examples to improve the generalization.Our main contributions are:• We build a connection between the generalization of models trained with noisy labels and the smoothness of solutions, which is related to the subspace dimensionality.• We propose a novel approach for training with noisy labels, which greatly mitigates overfitting. Our method is simple yet effective and can be applied to any neural network architecture. Additional knowledge on the clean validation dataset is not required.• A thorough empirical evaluation on various datasets (CIFAR-10, CIFAR-100) is conducted and demonstrates a significant improvement over the competing strong baselines. We propose a simple but effective algorithm for robust deep learning with noisy labels. Our method builds upon a variance regularizer that prevents the model from overfitting to the corrupted labels. Extensive experiments given in the paper show that the generalization performance of DNNs trained with corrupted labels can be improved significantly using our method, which can serve as a strong baseline for deep learning with noisy labels.
The paper proposed a simple yet effective baseline for learning with noisy labels.
{"src": "scitldr-AIC_train", "paper_id": "S1xnKi5BOV"}