To Aggregate or Not? Learning with Separate Noisy Labels
The rawly collected training data often comes with separate noisy labels
collected from multiple imperfect annotators (e.g., via crowdsourcing).
Typically one would first aggregate the separate noisy labels into one and
apply standard training methods. The literature has also studied extensively on
effective aggregation approaches. This paper revisits this choice and aims to
provide an answer to the question of whether one should aggregate separate
noisy labels into single ones or use them separately as given. We theoretically
analyze the performance of both approaches under the empirical risk
minimization framework for a number of popular loss functions, including the
ones designed specifically for the problem of learning with noisy labels. Our
theorems conclude that label separation is preferred over label aggregation
when the noise rates are high, or the number of labelers/annotations is
insufficient. Extensive empirical results validate our conclusion.