How to take multiple factors into account when evaluating a Game with a Purpose? How is player behaviour or participation influenced by different incentives? How does player engagement impact their accuracy in solving tasks? In this paper, we present a detailed investigation of multiple factors affecting the evaluation of a GWAP and we show how they impact on the achieved results. We inform our study with the experimental assessment of a GWAP designed to solve a multinomial classification task.
Other metrics to evaluate GWAPs can be borrowed from studies of social community @cite_20 and citizen science evolution @cite_23 ; in those cases, however, user participation's success'' is measured through simple indicators like number of participants and contributions, while a deeper investigation is needed to assess the effectiveness of participation. Behavioural studies in HCI research have investigated volunteer characterization in citizen science, defining engagement metrics and profiles @cite_3 @cite_10 , which may or may not apply to GWAP players.
@cite_10 @cite_3 @cite_23 @cite_20
2606977703 276677300 2033354470 1558395318
Citizen participation in online communities of scientific investigations has recently become more popular. Enhancing the engagement of citizens within these communities is a focus of attention for researchers and practitioners who want to amplify the impact on learning, science and society. This study investigates the relationship between engagement factors and behaviour patterns in an online community that requires high levels of citizen participation. While other studies explore engagement in communities where citizens contribute data, the current research investigates a community to support citizens in facilitating their own scientific investigations. Data were collected from log files and questionnaires, and multiple measures of engagement were examined: engagement metrics, roles, motivation, attitude, satisfaction and belonging to the community. The results allowed comparison of the engagement levels among different types of citizen participation communities and categorised members in engagement profiles, according to their behaviour patterns. Findings indicate a need for differing design approaches based on the type of citizen participation community and individual engagement profiles. Reliance on volunteer participation for citizen science has become extremely popular. Cutting across disciplines, locations, and participation practices, hundreds of thousands of volunteers throughout the world are helping scientists accomplish tasks they could not otherwise perform. Although existing projects have demonstrated the value of involving volunteers in data collection, relatively few projects have been successful in maintaining volunteers’ continued involvement over long periods of time. Therefore, it is important to understand the temporal nature of volunteers’ motivations and their effect on participation practices, so that effective partnerships between volunteers and scientists can be established. This paper presents case studies of longitudinal participation practices in citizen science in three countries—the United States, India, and Costa Rica. The findings reveal a temporal process of participation, in which initial participation stems in most cases from self-directed motivations, such as personal interest. In contrast, long-term participation is more complex and includes both self-directed motivations and collaborative motivations. In most online citizen science projects, a large proportion of participants contribute in small quantities. To investigate how low contributors differ from committed volunteers, we distributed a survey to members of the Old Weather project, followed by interviews with respondents selected according to a range of contribution levels. The studies reveal a complex relationship between motivations and contribution. Whilst high contributors were deeply engaged by social or competitive features, low contributors described a solitary experience of 'dabbling' in projects for short periods. Since the majority of participants exhibit this small-scale contribution pattern, there is great potential value in designing interfaces to tempt lone workers to complete 'just another page', or to lure early drop-outs back into participation. This includes breaking the work into components which can be tackled without a major commitment of time and effort, and providing feedback on the quality and value of these contributions. We present studies of the attention and time, or engagement, invested by crowd workers on tasks. Consideration of worker engagement is especially important in volunteer settings such as online citizen science. Using data from Galaxy Zoo, a prominent citizen science project, we design and construct statistical models that provide predictions about the forthcoming engagement of volunteers. We characterize the accuracy of predictions with respect to different sets of features that describe user behavior and study the sensitivity of predictions to variations in the amount of data and retraining. We design our model for guiding system actions in real-time settings, and discuss the prospect for harnessing predictive models of engagement to enhance user attention and effort on volunteer tasks.