A. Identifying Labelling Errors: We manually induce 10% noise by randomly flipping labels in the two public datasets (i.e. setup similar to [1]}). For private enterprise datasets (which correspond to real-world production data), we re-labelled the samples via MTurkhttps://www.mturk.com/, and found labelling errors in 1.5% and 33% of the input data, respectively. We observed that samples in the lower confidence and higher certainty region are further away from decision boundary, and correspond to hard or mislabelled training samples. The model found it difficult to learn from these samples potentially because they are noisy (refer Table 1 for examples).
