Eliciting and Learning with Soft Labels from Every Annotator
The labels used to train machine learning (ML) models are of paramount
importance. Typically for ML classification tasks, datasets contain hard
labels, yet learning using soft labels has been shown to yield benefits for
model generalization, robustness, and calibration. Earlier work found success
in forming soft labels from multiple annotators' hard labels; however, this
approach may not converge to the best labels and necessitates many annotators,
which can be expensive and inefficient. We focus on efficiently eliciting soft
labels from individual annotators. We collect and release a dataset of soft
labels for CIFAR-10 via a crowdsourcing study ($N=242$). We demonstrate that
learning with our labels achieves comparable model performance to prior
approaches while requiring far fewer annotators. Our elicitation methodology
therefore shows promise towards enabling practitioners to enjoy the benefits of
improved model performance and reliability with fewer annotators, and serves as
a guide for future dataset curators on the benefits of leveraging richer
information, such as categorical uncertainty, from individual annotators.