Due to concerns about human error in crowdsourcing, it is standard practice to collect labels for the same data point from multiple internet workers. We here show that the resulting budget can be used more effectively with a flexible worker assignment strategy that asks fewer workers to analyze easy-to-label data and more workers to analyze data that requires extra scrutiny. Our main contribution is to show how the allocations of the number of workers to a task can be computed optimally based on task features alone, without using worker profiles. Our target tasks are delineating cells in microscopy images and analyzing the sentiment toward the 2016 U.S. presidential candidates in tweets. We first propose an algorithm that computes budget-optimized crowd worker allocation (BUOCA). We next train a machine learning system (BUOCA-ML) that predicts an optimal number of crowd workers needed to maximize the accuracy of the labeling. We show that the computed allocation can yield large savings in the crowdsourcing budget (up to 49 percent points) while maintaining labeling accuracy. Finally, we envisage a human-machine system for performing budget-optimized data analysis at a scale beyond the feasibility of crowdsourcing.
Our work is distinct from prior work in that our system not only learns an optimal crowd worker allocation that is adapted to task difficulty, but also a mapping from data features to crowd worker allocations. In their award-winning paper, @cite_2 addressed a related data-focused problem -- how to solicit fewer human responses when answer agreement is expected and more responses otherwise, based on predicting from a visual question whether a crowd would agree on one answer. Their method computes the required budget after a classifier had been applied to rank the ambiguity in the data. Their system solicits at most five answers for the @math data points predicted to reflect the greatest likelihood for crowd disagreement and one answer for the remaining visual questions, where @math is the extra budget available. In our paradigm, the output of BUOCA determines the specific budget level when a sufficient number of data has been labeled so that the training of BUOCA-ML is expected to be successful. BUOCA-ML is then trained with the labels obtained with this training budget (note that the training budget is different from the budget needed to apply BUOCA-ML in phase 2).
@cite_2
2610525153
Visual question answering systems empower users to ask any question about any image and receive a valid answer. However, existing systems do not yet account for the fact that a visual question can lead to a single answer or multiple different answers. While a crowd often agrees, disagreements do arise for many reasons including that visual questions are ambiguous, subjective, or difficult. We propose a model, CrowdVerge, for automatically predicting from a visual question whether a crowd would agree on one answer. We then propose how to exploit these predictions in a novel application to efficiently collect all valid answers to visual questions. Specifically, we solicit fewer human responses when answer agreement is expected and more human responses otherwise. Experiments on 121,811 visual questions asked by sighted and blind people show that, compared to existing crowdsourcing systems, our system captures the same answer diversity with typically 14-23 less crowd involvement.