Deep neural networks trained on large supervised datasets have led to impressive results in recent years. However, since well-annotated datasets can be prohibitively expensive and time-consuming to collect, recent work has explored the use of larger but noisy datasets that can be more easily obtained. In this paper, we investigate the behavior of deep neural networks on training sets with massively noisy labels. We show on multiple datasets such as MINST, CIFAR-10 and ImageNet that successful learning is possible even with an essentially arbitrary amount of noise. For example, on MNIST we find that accuracy of above 90 percent is still attainable even when the dataset has been diluted with 100 noisy examples for each clean example. Such behavior holds across multiple patterns of label noise, even when noisy labels are biased towards confusing classes. Further, we show how the required dataset size for successful training increases with higher label noise. Finally, we present simple actionable techniques for improving learning in the regime of high label noise. Deep learning has proven to be powerful for a wide range of problems, from image classification to machine translation. Typically, deep neural networks are trained using supervised learning on large, carefully annotated datasets. However, the need for such datasets restricts the space of problems that can be addressed. This has led to a proliferation of deep learning results on the same tasks using the same well-known datasets. Carefully annotated data is difficult to obtain, especially for classification tasks with large numbers of classes (requiring extensive annotation) or with fine-grained classes (requiring skilled annotation). Thus, annotation can be expensive and, for tasks requiring expert knowledge, may simply be unattainable at scale.To address this limitation, other training paradigms have been investigated to alleviate the need for expensive annotations, such as unsupervised learning BID11 , self-supervised learning BID16 BID23 and learning from noisy annotations (Joulin et al., 2016; BID15 BID22 . Very large datasets (e.g., BID7 ; BID19 ) can often be attained, for example from web sources, with partial or unreliable annotation. This can allow neural networks to be trained on a much wider variety of tasks or classes and with less manual effort. The good performance obtained from these large noisy datasets indicates that deep learning approaches can tolerate modest amounts of noise in the training set.In this work, we take this trend to an extreme, and consider the performance of deep neural networks under extremely low label reliability, only slightly above chance. We envision a future in which arbitrarily large amounts of data will easily be obtained, but in which labels come without any guarantee of validity and may merely be biased towards the correct distribution.The key takeaways from this paper may be summarized as follows:• Deep neural networks are able to learn from data that has been diluted by an arbitrary amount of noise. We demonstrate that standard deep neural networks still perform well even on training sets in which label accuracy is as low as 1 percent above chance. On MNIST, for example, performance still exceeds 90 percent even with this level of label noise (see Figure 1 ). This behavior holds, to varying extents, across datasets as well as patterns of label noise, including when noisy labels are biased towards confused classes.• A sufficiently large training set can accommodate a wide range of noise levels. We find that the minimum dataset size required for effective training increases with the noise level. A large enough training set can accommodate a wide range of noise levels. Increasing the dataset size further, however, does not appreciably increase accuracy.• Adjusting batch size and learning rate can allow conventional neural networks to operate in the regime of very high label noise. We find that label noise reduces the effective batch size, as noisy labels roughly cancel out and only a small learning signal remains. We show that dataset noise can be partly compensated for by larger batch sizes and by scaling the learning rate with the effective batch size. In this paper, we have considered the behavior of deep neural networks on training sets with very noisy labels. In a series of experiments, we have demonstrated that learning is robust to an essentially arbitrary amount of label noise, provided that the number of clean labels is sufficiently large. We have further shown that the threshold required for clean labels increases as the noise level does. Finally, we have observed that noisy labels reduce the effective batch size, an effect that can be mitigated by larger batch sizes and downscaling the learning rate.It is worthy of note that although deep networks appear robust to even high degrees of label noise, clean labels still always perform better than noisy labels, given the same quantity of training data. Further, one still requires expert-vetted test sets for evaluation. Lastly, it is important to reiterate that our studies focus on non-adversarial noise.Our work suggests numerous directions for future investigation. For example, we are interested in how label-cleaning and semi-supervised methods affect the performance of networks in a high-noise regime. Are such approaches able to lower the threshold for training set size? Finally, it remains to translate the results we present into an actionable trade-off between data annotation and acquisition costs, which can be utilized in real world training pipelines for deep networks on massive noisy data.
We show that deep neural networks are able to learn from data that has been diluted by an arbitrary amount of noise.
{"src": "scitldr-AIC_train", "paper_id": "B1p461b0W"}