How to take multiple factors into account when evaluating a Game with a Purpose? How is player behaviour or participation influenced by different incentives? How does player engagement impact their accuracy in solving tasks? In this paper, we present a detailed investigation of multiple factors affecting the evaluation of a GWAP and we show how they impact on the achieved results. We inform our study with the experimental assessment of a GWAP designed to solve a multinomial classification task.
Participation incentives are usually classified as intrinsic or extrinsic motivation @cite_4 . Some comparative analysis of incentives exists for GWAPs @cite_5 , especially in contrast to different methods like micro-working @cite_8 @cite_26 @cite_14 or machine learning @cite_18 . The effect of competition and tangible rewards on participation and quality of results has also been explored, both in the context of GWAPs @cite_21 and online citizen science campaigns @cite_22 , revealing the pros and cons of designing different motivation mechanisms.
@cite_18 @cite_14 @cite_26 @cite_4 @cite_22 @cite_8 @cite_21 @cite_5
2058911993 2039922069 2088940490 2124576904 2005406065 2041411104 2170493558 1532110328
User participation is one of the most important elements in participatory sensing application for providing adequate level of service quality. However, incentive mechanism and its economic model for user participation have been less addressed so far in this research domain. This paper studies the economic model of user participation incentive in participatory sensing applications. To stimulate user participation, we design and evaluate a novel reverse auction based dynamic pricing incentive mechanism where users can sell their sensing data to a service provider with users' claimed bid prices. The proposed incentive mechanism focuses on minimizing and stabilizing the incentive cost while maintaining adequate level of participants by preventing users from dropping out of participatory sensing applications. Compared with random selection based fixed pricing incentive mechanism, the proposed mechanism not only reduces the incentive cost for retaining the same number of participants but also improves the fairness of incentive distribution and social welfare. It also helps us to achieve the geographically balanced sensing measurements and, more importantly, can remove the burden of accurate price decision for user data that is the most difficult step in designing incentive mechanism. In crowdsensing, appropriate rewards are always expected to compensate the participants for their consumptions of physical resources and involvements of manual efforts. While continuous low quality sensing data could do harm to the availability and preciseness of crowdsensing based services, few existing incentive mechanisms have ever addressed the issue of sensing data's quality. The design of quality based incentive mechanism is motivated by its potential to avoid inefficient sensing and unnecessary rewards. In this paper, we incorporate the consideration of data quality into the design of incentive mechanism for crowdsensing, and propose to pay the participants as how well they do, to motivate the rational participants to perform data sensing efficiently. This mechanism estimates the quality of sensing data, and offers each participant a reward based on her effective contribution. We also implement the mechanism and evaluate the improvements in terms of quality of service and profit of service provider. The evaluation results show that our mechanism achieves superior performance when compared to the uniform pricing scheme. This paper studies economic models of user participation incentive in participatory sensing applications. User participation is the most important element in participatory sensing applications for providing adequate level of service quality. However, incentive mechanism and its economic model for user participation have never been addressed so far in this research domain. In order to stimulate user participation, we design and evaluate a novel Reverse Auction based Dynamic Price (RADP) incentive mechanism, where users can sell their sensing data to a service provider with users' claimed bid prices. The proposed incentive mechanism focuses on minimizing and stabilizing incentive cost while maintaining adequate number of participants by preventing users from dropping out of participatory sensing applications. Compared with a Random Selection with Fixed Price (RSFP) incentive mechanism, the proposed mechanism not only reduces the incentive cost for retaining same number of participants by more than 60 but also improves the fairness of incentive distribution and social welfare. More importantly, RADP can remove burden of accurate pricing for user sensing data, the most difficult step in RSFP. In this paper we present and analyze a model in which users select among, and subsequently compete in, a collection of contests offering various rewards. The objective is to capture the essential features of a crowdsourcing system, an environment in which diverse tasks are presented to a large community. We aim to demonstrate the precise relationship between incentives and participation in such systems. We model contests as all-pay auctions with incomplete information; as a consequence of revenue equivalence, our model may also be interpreted more broadly as one in which users select among auctions of heterogeneous goods. We present two regimes in which we find an explicit correspondence in equilibrium between the offered rewards and the users' participation levels. The regimes respectively model situations in which different contests require similar or unrelated skills. Principally, we find that rewards yield logarithmically diminishing returns with respect to participation levels. We compare these results to empirical data from the crowdsourcing site Taskcn.com; we find that as we condition the data on more experienced users, the model more closely conforms to the empirical data. Participatory sensing has emerged as a novel paradigm for data collection and collective knowledge formation about a state or condition of interest, sometimes linked to a geographic area. In this paper, we address the problem of incentive mechanism design for data contributors for participatory sensing applications. The service provider receives service queries in an area from service requesters and initiates an auction for user participation. Upon request, each user reports its perceived cost per unit of amount of participation, which essentially maps to a requested amount of compensation for participation. The participation cost quantifies the dissatisfaction caused to user due to participation. This cost is considered to be private information for each device, as it strongly depends on various factors inherent to it, such as the energy cost for sensing, data processing and transmission to the closest point of wireless access, the residual battery level, the number of concurrent jobs at the device processor, the required bandwidth to transmit data and the related charges of the mobile network operator, or even the user discomfort due to manual effort to submit data. Hence, participants have strong motive to mis-report their cost, i.e. declare a higher cost that the actual one, so as to obtain higher payment. We seek a mechanism for user participation level determination and payment allocation which is most viable for the provider, that is, it minimizes the total cost of compensating participants, while delivering a certain quality of experience to service requesters. We cast the problem in the context of optimal reverse auction design, and we show how the different quality of submitted information by participants can be tracked by the service provider and used in the participation level and payment selection procedures. We derive a mechanism that optimally solves the problem above, and at the same time it is individually rational (i.e., it motivates users to participate) and incentive-compatible (i.e. it motivates truthful cost reporting by participants). Finally, a representative participatory sensing case study involving parameter estimation is presented, which exemplifies the incentive mechanism above. The relationship between financial incentives and performance, long of interest to social scientists, has gained new relevance with the advent of web-based "crowd-sourcing" models of production. Here we investigate the effect of compensation on performance in the context of two experiments, conducted on Amazon's Mechanical Turk (AMT). We find that increased financial incentives increase the quantity, but not the quality, of work performed by participants, where the difference appears to be due to an "anchoring" effect: workers who were paid more also perceived the value of their work to be greater, and thus were no more motivated than workers paid less. In contrast with compensation levels, we find the details of the compensation scheme do matter---specifically, a "quota" system results in better work for less pay than an equivalent "piece rate" system. Although counterintuitive, these findings are consistent with previous laboratory studies, and may have real-world analogs as well. We study the causal effects of financial incentives on the quality of crowdwork. We focus on performance-based payments (PBPs), bonus payments awarded to workers for producing high quality work. We design and run randomized behavioral experiments on the popular crowdsourcing platform Amazon Mechanical Turk with the goal of understanding when, where, and why PBPs help, identifying properties of the payment, payment structure, and the task itself that make them most effective. We provide examples of tasks for which PBPs do improve quality. For such tasks, the effectiveness of PBPs is not too sensitive to the threshold for quality required to receive the bonus, while the magnitude of the bonus must be large enough to make the reward salient. We also present examples of tasks for which PBPs do not improve quality. Our results suggest that for PBPs to improve quality, the task must be effort-responsive: the task must allow workers to produce higher quality work by exerting more effort. We also give a simple method to determine if a task is effort-responsive a priori. Furthermore, our experiments suggest that all payments on Mechanical Turk are, to some degree, implicitly performance-based in that workers believe their work may be rejected if their performance is sufficiently poor. Finally, we propose a new model of worker behavior that extends the standard principal-agent model from economics to include a worker's subjective beliefs about his likelihood of being paid, and show that the predictions of this model are in line with our experimental findings. This model may be useful as a foundation for theoretical studies of incentives in crowdsourcing markets. Providing an adequate long-term user participation incentive is important for a participatory sensing system to maintain enough number of active users (sensors), so as to collect a sufficient number of data samples and support a desired level of service quality. In this work, we consider the sensor selection problem in a general time-dependent and location-aware participatory sensing system, taking the long-term user participation incentive into explicit consideration. We study the problem systematically under different information scenarios, regarding both future information and current information (realization). In particular, we propose a Lyapunov-based VCG auction policy for the on-line sensor selection, which converges asymptotically to the optimal off-line benchmark performance, even with no future information and under asymmetry of current information. Extensive numerical results show that our proposed policy outperforms the state-of-art policies in the literature, in terms of both user participation (e.g., reducing the user dropping probability by 25 ∼ 90 ) and social performance (e.g., increasing the social welfare by 15 ∼ 80 ).