An important class of game-theoretic incentive mechanisms for eliciting effort from a crowd are the peer based mechanisms, in which workers are paid by matching their answers with one another. The other classic mechanism is to have the workers solve some gold standard tasks and pay them according to their accuracy on gold tasks. This mechanism ensures stronger incentive compatibility than the peer based mechanisms but assigning gold tasks to all workers becomes inefficient at large scale. We propose a novel mechanism that assigns gold tasks to only a few workers and exploits transitivity to derive accuracy of the rest of the workers from their peers' accuracy. We show that the resulting mechanism ensures a dominant notion of incentive compatibility and fairness.
Finally, the mechanism of @cite_4 combines ideas from the two categories of work. It arranges the workers in a hierarchy. A constant number of workers in the top level of the hierarchy are evaluated by an oracle. The workers below that level are evaluated by the workers (peers) in one level above them. The mechanism solves the scalability issue of the gold tasks based mechanisms. Though it offers comparatively weaker incentive compatibility (unique Nash equilibrium) as compared to the gold tasks based mechanisms but eliminates all the undesired equilibria that exist in peer based mechanisms. However, it requires that workers are informed of their level in the hierarchy. The mechanism is also ex-ante unfair towards the workers in the sense that workers in the top level of hierarchy are evaluated more correctly than the workers in the lower levels. Similar to @cite_4 , our work is also at the intersection of the two categories of works. However, our mechanism doesn't suffer from the issues (level information requirement and unfairness) that their mechanism has and also guarantees stronger incentive compatibility.
@cite_4
2514819328
We consider crowdsourcing problems where the users are asked to provide evaluations for items; the user evaluations are then used directly, or aggregated into a consensus value. Lacking an incentive scheme, users have no motive in making effort in completing the evaluations, providing inaccurate answers instead. We propose incentive schemes that are truthful and cheap: truthful as the optimal user behavior consists in providing accurate evaluations, and cheap because the truthfulness is achieved with little overhead cost. We consider both discrete evaluation tasks, where an evaluation can be done either correctly, or incorrectly, with no degrees of approximation in between, and quantitative evaluation tasks, where evaluations are real numbers, and the error is measured as distance from the correct value. For both types of tasks, we propose hierarchical incentive schemes that can be effected with a small amount of additional evaluations, and that scale to arbitrarily large crowd sizes: they have the property that the strength of the incentive does not weaken with increasing hierarchy depth. Interestingly, we show that for these schemes to work, the only requisite is that workers know their place in the hierarchy in advance.