Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> Distributing labeling tasks among hundreds or thousands of annotators is an increasingly important method for annotating large datasets. We present a method for estimating the underlying value (e.g. the class) of each image from (noisy) annotations provided by multiple annotators. Our method is based on a model of the image formation and annotation process. Each image has different characteristics that are represented in an abstract Euclidean space. Each annotator is modeled as a multidimensional entity with variables representing competence, expertise and bias. This allows the model to discover and represent groups of annotators that have different sets of skills and knowledge, as well as groups of images that differ qualitatively. We find that our model predicts ground truth labels on both synthetic and real data more accurately than state of the art methods. Experiments also show that our model, starting from a set of binary labels, may discover rich information, such as different "schools of thought" amongst the annotators, and can group together images belonging to separate categories. <s> BIB001 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> This article proposes a consistent and efficient estimator of the high-frequency covariance (quadratic covariation) of two arbitrary assets, observed asynchronously with market microstructure noise. This estimator is built on the marriage of the quasi–maximum likelihood estimator of the quadratic variation and the proposed generalized synchronization scheme and thus is not influenced by the Epps effect. Moreover, the estimation procedure is free of tuning parameters or bandwidths and is readily implementable. Monte Carlo simulations show the advantage of this estimator by comparing it with a variety of estimators with specific synchronization methods. The empirical studies of six foreign exchange future contracts illustrate the time-varying correlations of the currencies during the 2008 global financial crisis, demonstrating the similarities and differences in their roles as key currencies in the global market. <s> BIB002 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> In this paper, we discuss how we can extend probabilistic topic models to analyze the relationship graph of popular social-network data, so that we can group or label the edges and nodes in the graph based on their topic similarity. In particular, we first apply the well-known Latent Dirichlet Allocation (LDA) model and its existing variants to the graph-labeling task and argue that the existing models do not handle popular nodes (nodes with many incoming edges) in the graph very well. We then propose possible extensions to this model to deal with popular nodes. Our experiments show that the proposed extensions are very effective in labeling popular nodes, showing significant improvements over the existing methods. Our proposed methods can be used for providing, for instance, more relevant friend recommendations within a social network. <s> BIB003 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> The research work presented here includes data mining needs and study of their algorithm for various extraction purpose. It also includes work that has been done in the field of harvesting images from web. Here the proposed method is to harvest image databases from web. We can automatically generate a large number of images for a specified object. By applying concept of data mining and the algorithm from data mining which is used for extraction of data or harvesting images. A multimodal approach employing text ,metadata and visual features is used to gather many high-quality images from the web. The modules can be made to find query images by selecting images where nearby text is top ranked by the topic i.e., formation of image clusters then download associate images by using approaches like web search, image search and Google images. Apply re-ranking algorithm and then filtering process to harvest the images.Currently, image search gives a very low precision (only about 4%) and is not used for the harvesting experiments. Since the movements of the technologies are growing rapidly the kinds of work also need to be grown up. This work shows an approach to harvest a large number of images of a particular class automatically and to achieve this with high precision by providing training databases so that a new object model can be learned effortlessly. Many other tools also are available for harvesting images from web .An approach in this paper is original and up to the mark. Keywords: Legacy code, re-engineering, class diagrams, Aggregation, Association, Attributes <s> BIB004 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> Current approaches for fine-grained recognition do the following: First, recruit experts to annotate a dataset of images, optionally also collecting more structured data in the form of part annotations and bounding boxes. Second, train a model utilizing this data. Toward the goal of solving fine-grained recognition, we introduce an alternative approach, leveraging free, noisy data from the web and simple, generic methods of recognition. This approach has benefits in both performance and scalability. We demonstrate its efficacy on four fine-grained datasets, greatly exceeding existing state of the art without the manual collection of even a single label, and furthermore show first results at scaling to more than 10,000 fine-grained categories. Quantitatively, we achieve top-1 accuracies of \(92.3\,\%\) on CUB-200-2011, \(85.4\,\%\) on Birdsnap, \(93.4\,\%\) on FGVC-Aircraft, and \(80.8\,\%\) on Stanford Dogs without using their annotated training sets. We compare our approach to an active learning approach for expanding fine-grained datasets. <s> BIB005 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> Importance ::: Deep learning is a family of computational methods that allow an algorithm to program itself by learning from a large set of examples that demonstrate the desired behavior, removing the need to specify rules explicitly. Application of these methods to medical imaging requires further assessment and validation. ::: ::: ::: Objective ::: To apply deep learning to create an algorithm for automated detection of diabetic retinopathy and diabetic macular edema in retinal fundus photographs. ::: ::: ::: Design and Setting ::: A specific type of neural network optimized for image classification called a deep convolutional neural network was trained using a retrospective development data set of 128 175 retinal images, which were graded 3 to 7 times for diabetic retinopathy, diabetic macular edema, and image gradability by a panel of 54 US licensed ophthalmologists and ophthalmology senior residents between May and December 2015. The resultant algorithm was validated in January and February 2016 using 2 separate data sets, both graded by at least 7 US board-certified ophthalmologists with high intragrader consistency. ::: ::: ::: Exposure ::: Deep learning-trained algorithm. ::: ::: ::: Main Outcomes and Measures ::: The sensitivity and specificity of the algorithm for detecting referable diabetic retinopathy (RDR), defined as moderate and worse diabetic retinopathy, referable diabetic macular edema, or both, were generated based on the reference standard of the majority decision of the ophthalmologist panel. The algorithm was evaluated at 2 operating points selected from the development set, one selected for high specificity and another for high sensitivity. ::: ::: ::: Results ::: The EyePACS-1 data set consisted of 9963 images from 4997 patients (mean age, 54.4 years; 62.2% women; prevalence of RDR, 683/8878 fully gradable images [7.8%]); the Messidor-2 data set had 1748 images from 874 patients (mean age, 57.6 years; 42.6% women; prevalence of RDR, 254/1745 fully gradable images [14.6%]). For detecting RDR, the algorithm had an area under the receiver operating curve of 0.991 (95% CI, 0.988-0.993) for EyePACS-1 and 0.990 (95% CI, 0.986-0.995) for Messidor-2. Using the first operating cut point with high specificity, for EyePACS-1, the sensitivity was 90.3% (95% CI, 87.5%-92.7%) and the specificity was 98.1% (95% CI, 97.8%-98.5%). For Messidor-2, the sensitivity was 87.0% (95% CI, 81.1%-91.0%) and the specificity was 98.5% (95% CI, 97.7%-99.1%). Using a second operating point with high sensitivity in the development set, for EyePACS-1 the sensitivity was 97.5% and specificity was 93.4% and for Messidor-2 the sensitivity was 96.1% and specificity was 93.9%. ::: ::: ::: Conclusions and Relevance ::: In this evaluation of retinal fundus photographs from adults with diabetes, an algorithm based on deep machine learning had high sensitivity and specificity for detecting referable diabetic retinopathy. Further research is necessary to determine the feasibility of applying this algorithm in the clinical setting and to determine whether use of the algorithm could lead to improved care and outcomes compared with current ophthalmologic assessment. <s> BIB006 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> Recommendation and collaborative filtering systems are important in modern information and e-commerce applications. As these systems are becoming increasingly popular in the industry, their outputs could affect business decision making, introducing incentives for an adversarial party to compromise the availability or integrity of such systems. We introduce a data poisoning attack on collaborative filtering systems. We demonstrate how a powerful attacker with full knowledge of the learner can generate malicious data so as to maximize his/her malicious objectives, while at the same time mimicking normal user behavior to avoid being detected. While the complete knowledge assumption seems extreme, it enables a robust assessment of the vulnerability of collaborative filtering schemes to highly motivated attacks. We present efficient solutions for two popular factorization-based collaborative filtering algorithms: the \emph{alternative minimization} formulation and the \emph{nuclear norm minimization} method. Finally, we test the effectiveness of our proposed algorithms on real-world data and discuss potential defensive strategies. <s> BIB007 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> Data are often labeled by many different experts with each expert only labeling a small fraction of the data and each data point being labeled by several experts. This reduces the workload on individual experts and also gives a better estimate of the unobserved ground truth. When experts disagree, the standard approaches are to treat the majority opinion as the correct label or to model the correct label as a distribution. These approaches, however, do not make any use of potentially valuable information about which expert produced which label. To make use of this extra information, we propose modeling the experts individually and then learning averaging weights for combining them, possibly in sample-specific ways. This allows us to give more weight to more reliable experts and take advantage of the unique strengths of individual experts at classifying certain types of data. Here we show that our approach leads to improvements in computer-aided diagnosis of diabetic retinopathy. We also show that our method performs better than competing algorithms by Welinder and Perona (2010), and by Mnih and Hinton (2012). Our work offers an innovative approach for dealing with the myriad real-world settings that use expert opinions to define labels for training. <s> BIB008 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> A weakly supervised semantic segmentation (WSSS) method aims to learn a segmentation model from weak (image-level) as opposed to strong (pixel-level) labels. By avoiding the tedious pixel-level annotation process, it can exploit the unlimited supply of user-tagged images from media-sharing sites such as Flickr for large scale applications. However, these ‘free’ tags/labels are often noisy and few existing works address the problem of learning with both weak and noisy labels. In this work, we cast the WSSS problem into a label noise reduction problem. Specifically, after segmenting each image into a set of superpixels, the weak and potentially noisy image-level labels are propagated to the superpixel level resulting in highly noisy labels; the key to semantic segmentation is thus to identify and correct the superpixel noisy labels. To this end, a novel $L_1$ -optimisation based sparse learning model is formulated to directly and explicitly detect noisy labels. To solve the $L_1$ -optimisation problem, we further develop an efficient learning algorithm by introducing an intermediate labelling variable. Extensive experiments on three benchmark datasets show that our method yields state-of-the-art results given noise-free labels, whilst significantly outperforming the existing methods when the weak labels are also noisy. <s> BIB009 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> Supervised learning depends on annotated examples, which are taken to be the \emph{ground truth}. But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk. Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem). Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples. This raises two fundamental questions: (1) How can we best learn from noisy workers? (2) How should we allocate our labeling budget to maximize the performance of a classifier? We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data. The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality. Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality. We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality is above a threshold. Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits. <s> BIB010 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> Machine learning systems trained on user-provided data are susceptible to data poisoning attacks, whereby malicious users inject false training data with the aim of corrupting the learned model. While recent work has proposed a number of attacks and defenses, little is understood about the worst-case loss of a defense in the face of a determined attacker. We address this by constructing approximate upper bounds on the loss across a broad family of attacks, for defenders that first perform outlier removal followed by empirical risk minimization. Our bound comes paired with a candidate attack that nearly realizes the bound, giving us a powerful tool for quickly assessing defenses on a given dataset. Empirically, we find that even under a simple defense, the MNIST-1-7 and Dogfish datasets are resilient to attack, while in contrast the IMDB sentiment dataset can be driven from 12% to 23% test error by adding only 3% poisoned data. <s> BIB011 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> Building classification models from clinical data often requires human experts for example labeling. However, it is difficult to obtain a perfect set of labels due to the complexity of the medical data and the large variability between experts. In this study we present a neural-network training strategy that is more robust to unreliable labeling by explicitly modeling the label noise as part of the network architecture. Our method is demonstrated on breast microcalcifications classification into benign and malignant categories, given multi-view mammograms. We show that the proposed training procedure outperforms standard training methods that ignore the existence of label noise. <s> BIB012 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> The volume and complexity of diagnostic imaging is increasing at a pace faster than the availability of human expertise to interpret it. Artificial intelligence has shown great promise in classifying two-dimensional photographs of some common diseases and typically relies on databases of millions of annotated images. Until now, the challenge of reaching the performance of expert clinicians in a real-world clinical pathway with three-dimensional diagnostic scans has remained unsolved. Here, we apply a novel deep learning architecture to a clinically heterogeneous set of three-dimensional optical coherence tomography scans from patients referred to a major eye hospital. We demonstrate performance in making a referral recommendation that reaches or exceeds that of experts on a range of sight-threatening retinal diseases after training on only 14,884 scans. Moreover, we demonstrate that the tissue segmentations produced by our architecture act as a device-independent representation; referral accuracy is maintained when using tissue segmentations from a different type of device. Our work removes previous barriers to wider clinical use without prohibitive training data requirements across multiple pathologies in a real-world setting. <s> BIB013 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> Deep neural networks (DNNs) have achieved great success in a wide variety of medical image analysis tasks. However, these achievements indispensably rely on the accurately-annotated datasets. If with the noisy-labeled images, the training procedure will immediately encounter difficulties, leading to a suboptimal classifier. This problem is even more crucial in the medical field, given that the annotation quality requires great expertise. In this paper, we propose an effective iterative learning framework for noisy-labeled medical image classification, to combat the lacking of high quality annotated medical data. Specifically, an online uncertainty sample mining method is proposed to eliminate the disturbance from noisy-labeled images. Next, we design a sample re-weighting strategy to preserve the usefulness of correctly-labeled hard samples. Our proposed method is validated on skin lesion classification task, and achieved very promising results. <s> BIB014 </s> Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey <s> C. Sources of Label Noise <s> We tackle the problem of semantic boundary prediction, which aims to identify pixels that belong to object(class) boundaries. We notice that relevant datasets consist of a significant level of label noise, reflecting the fact that precise annotations are laborious to get and thus annotators trade-off quality with efficiency. We aim to learn sharp and precise semantic boundaries by explicitly reasoning about annotation noise during training. We propose a simple new layer and loss that can be used with existing learning-based boundary detectors. Our layer/loss enforces the detector to predict a maximum response along the normal direction at an edge, while also regularizing its direction. We further reason about true object boundaries during training using a level set formulation, which allows the network to learn from misaligned labels in an end-to-end fashion. Experiments show that we improve over the CASENet backbone network by more than 4% in terms of MF(ODS) and 18.61% in terms of AP, outperforming all current state-of-the-art methods including those that deal with alignment. Furthermore, we show that our learned network can be used to significantly improve coarse segmentation labels, lending itself as an efficient way to label new data. <s> BIB015
As mentioned, label noise is a natural outcome of dataset collection process and can be seen in various domains, such as medical imaging BIB008 , BIB012 , BIB014 , semantic segmentation BIB009 - BIB015 , crowd-sourcing BIB001 , social network tagging BIB003 , financial analysis BIB002 and many more. This work focuses on the solutions to such problems, but it may be helpful to investigate the causes of label noise to better understand the phenomenon. Firstly, with the availability of gigantic amount of data on web and social media, it is a great interest of computer vision community to make use of that BIB004 - BIB005 . However labels of these data are coming from messy user tags or automated systems used by search engines. These processes of obtaining dataset are well known to result in noisy labels. Secondly, dataset can be labeled by multiple experts resulting in multi-labeled dataset. Each labeler has varying level of expertise and their opinions may commonly conflict with each other, which results in noisy label problem BIB010 . There are several reasons to get data labeled by more than one expert. Opinions of multiple labelers can be used to double check each others predictions for difficult datasets or crowd-sourcing platforms can be used to decrease the cost of labeling for big data; such as Amazon Mechanical Turk, Crowdflower and more. Despite its cheapness, labels obtained from non experts are commonly noisy with differentiating rate of error. Some labelers even can be a total spammer whose labels are random noise BIB010 . Thirdly, data can be too complex for even the experts in the field, e.g medical imaging. For example, to collect gold standard validation data for retinal images, annotations are gathered from 6-8 different experts BIB013 , BIB006 . This can be due to subjectiveness of the task for human experts or the lack of experience in annotator. Considering the fields where the true diagnosis is of crucial importance, overcoming this noise can be of great interest. Lastly, adversarial label noise can be inserted for the purpose of data poisoning to harm the learning procedure BIB007 , BIB011 .