Since deep neural networks are over-parameterized, they can memorize noisy examples. We address such memorizing issue in the presence of annotation noise. From the fact that deep neural networks cannot generalize neighborhoods of the features acquired via memorization, we hypothesize that noisy examples do not consistently incur small losses on the network under a certain perturbation. Based on this, we propose a novel training method called Learning with Ensemble Consensus (LEC) that prevents overfitting noisy examples by eliminating them using the consensus of an ensemble of perturbed networks. One of the proposed LECs, LTEC outperforms the current state-of-the-art methods on noisy MNIST, CIFAR-10, and CIFAR-100 in an efficient manner. Deep neural networks (DNNs) have shown excellent performance (Krizhevsky et al., 2012; He et al., 2016) on visual recognition datasets (Deng et al., 2009) . However, it is difficult to obtain highquality labeled datasets in practice (Wang et al., 2018a) . Even worse, DNNs could not generalize the training data in the presence of noisy examples . Therefore, there is an increasing demand for robust training methods. In general, DNNs optimized with SGD first generalize clean examples under label noise . Based on this, recent studies consider examples that incur small losses on the network that does not overfit noisy examples as being clean (Han et al., 2018; Shen & Sanghavi, 2019) . However, such small-loss examples may be corrupted, particularly under a high level of noise. Hence, choosing safe examples from the noisy dataset with small-loss criteria may be impractical. To address this, we find the method of screening out noisy examples among small-loss examples by focusing on well-known observations: (i) noisy examples are learned via memorization rather than via generalization and (ii) under a certain perturbation, network predictions for memorized features easily fluctuate, while those for generalized features do not. Based on these two observations, we hypothesize that out of small-loss examples, training losses of noisy examples would increase by injecting certain perturbation to network parameters, while those of clean examples would not. This suggests that examples that consistently incur small losses under multiple perturbations can be considered as being clean. Since this idea comes from an artifact of SGD optimization, it can be applied to any architecture optimized with SGD. In this work, we introduce a method of perturbing parameters to filter noisy examples out of smallloss examples. By embedding the filtering into training, we propose a new robust training scheme termed learning with ensemble consensus (LEC). In LEC, the network is first trained on the entire training set for a while and then trained on the intersection of small-loss examples of the ensemble of perturbed networks. We present three LECs with different perturbations and evaluate their effectiveness on three benchmark datasets with random label noise (Goldberger & Ben-Reuven, 2016; Ma et al., 2018) , open-set noise (Wang et al., 2018b) , and semantic noise. The proposed LEC outperforms existing robust training methods by efficiently removing noisy examples from training batches. Generalization of DNNs. Although DNNs are over-parameterized, they have impressive generalization ability (Krizhevsky et al., 2012; He et al., 2016) . Some studies argue that gradient-based optimization plays an important role in regularizing DNNs (Neyshabur et al., 2014; . show that DNNs optimized with gradient-based methods generalize clean examples in the early stage of training. Since mislabeling reduces the correlation with other training examples, it is likely that noisy examples are learned via memorization. Therefore, we analyze the difference between generalized and memorized features to discriminate clean and noisy examples. Training DNNs with Noisy datasets. Label noise issues can be addressed by reducing negative impact of noisy examples. One direction is to train with a modified loss function based on the noise distribution. Most studies of this direction estimate the noise distribution prior to training as it is not accessible in general (Sukhbaatar et al., 2014; Goldberger & Ben-Reuven, 2016; Patrini et al., 2017; Hendrycks et al., 2018) . Another direction is to train with modified labels using the current model prediction (Reed et al., 2014; Ma et al., 2018) . Aside from these directions, recent work suggests a method of exploiting small-loss examples (Jiang et al., 2017; Han et al., 2018; Yu et al., 2019; Shen & Sanghavi, 2019) based on the generalization ability of DNNs. However, it is still hard to find clean examples by relying on training losses. This study presents a simple method to overcome such a problem of small-loss criteria. This work presents the method of generating and using the ensemble for robust training. We explore three simple perturbation methods to generate the ensemble and then develop the way of identifying noisy examples through ensemble consensus on small-loss examples. Along with growing attention to the use of small-loss examples for robust training, we expect that our ensemble method will be useful for such training methods.
This work presents a method of generating and using ensembles effectively to identify noisy examples in the presence of annotation noise. 
{"src": "scitldr-AIC_train", "paper_id": "ryxOUTVYDH"}