We present the results of a user study with novice NMR analysts (N=19) involving a gamified simulation of the NMR analysis process. Participants solved randomly generated spectrum puzzles for up to three hours. We used eye tracking, event logging, and observations to record symptoms of cognitive depletion while participants worked. Analysis of results indicate that we can detect both signs of learning and signs of cognitive depletion in participants over the course of the three hours. Participants' break strategies did not predict or reflect game scores, but certain symptoms appear predictive of breaks.
The study of Mechanical Turk-style economies has provided a wealth of techniques for assessing the quality of worker contributions and for detecting workers who are abusing task structures for their own gain @cite_21 @cite_3 @cite_8 . These techniques are beneficial to the study of cognitive depletion as they provide potential metrics that can be automatically collected without interrupting workers as well as providing insight into the working patterns of large groups of individuals. It also provides an easily accessible real-world example of an ideal scenario for cognitive depletion research: an economy based on constant completion of micro-tasks where workers are motivated to work beyond the point where their cognitively depleted state begins affecting the quality of their output.
@cite_21 @cite_3 @cite_8
1999308248 2124994029 2030192188
Crowdsourcing systems lack effective measures of the effort required to complete each task. Without knowing how much time workers need to execute a task well, requesters struggle to accurately structure and price their work. Objective measures of effort could better help workers identify tasks that are worth their time. We propose a data-driven effort metric, ETA (error-time area), that can be used to determine a task's fair price. It empirically models the relationship between time and error rate by manipulating the time that workers have to complete a task. ETA reports the area under the error-time curve as a continuous metric of worker effort. The curve's 10th percentile is also interpretable as the minimum time most workers require to complete the task without error, which can be used to price the task. We validate the ETA metric on ten common crowdsourcing tasks, including tagging, transcription, and search, and find that ETA closely tracks how workers would rank these tasks by effort. We also demonstrate how ETA allows requesters to rapidly iterate on task designs and measure whether the changes improve worker efficiency. Our findings can facilitate the process of designing, pricing, and allocating crowdsourcing tasks. Detecting and correcting low quality submissions in crowdsourcing tasks is an important challenge. Prior work has primarily focused on worker outcomes or reputation, using approaches such as agreement across workers or with a gold standard to evaluate quality. We propose an alternative and complementary technique that focuses on the way workers work rather than the products they produce. Our technique captures behavioral traces from online crowd workers and uses them to predict outcome measures such quality, errors, and the likelihood of cheating. We evaluate the effectiveness of the approach across three contexts including classification, generation, and comprehension tasks. The results indicate that we can build predictive models of task performance based on behavioral traces alone, and that these models generalize to related tasks. Finally, we discuss limitations and extensions of the approach. Multitasking in user behavior can be represented along a continuum in terms of the time spent on one task before switching to another. In this paper, we present a theory of behavior along the multitasking continuum, from concurrent tasks with rapid switching to sequential tasks with longer time between switching. Our theory unifies several theoretical effects - the ACT-R cognitive architecture, the threaded cognition theory of concurrent multitasking, and the memory-for-goals theory of interruption and resumption - to better understand and predict multitasking behavior. We outline the theory and discuss how it accounts for numerous phenomena in the recent empirical literature.