Unsupervised Crowdsourcing with Accuracy and Cost Guarantees
We consider the problem of cost-optimal utilization of a crowdsourcing
platform for binary, unsupervised classification of a collection of items,
given a prescribed error threshold. Workers on the crowdsourcing platform are
assumed to be divided into multiple classes, based on their skill, experience,
and/or past performance. We model each worker class via an unknown confusion
matrix, and a (known) price to be paid per label prediction. For this setting,
we propose algorithms for acquiring label predictions from workers, and for
inferring the true labels of items. We prove that if the number of (unlabeled)
items available is large enough, our algorithms satisfy the prescribed error
thresholds, incurring a cost that is near-optimal. Finally, we validate our
algorithms, and some heuristics inspired by them, through an extensive case
study.