TITLE: Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement 

ABSTRACT: Machine Learning models have many potentially beneficial applications in
education settings, but a key barrier to their development is securing enough
data to train these models. Labelling educational data has traditionally relied
on highly skilled raters using complex, multi-class rubrics, making the process
expensive and difficult to scale. An alternative, more scalable approach could
be to use non-expert crowdworkers to evaluate student work, however,
maintaining sufficiently high levels of accuracy and inter-rater reliability
when using non-expert workers is challenging. This paper reports on two
experiments investigating using non-expert crowdworkers and comparative
judgement to evaluate complex student data. Crowdworkers were hired to evaluate
student responses to open-ended reading comprehension questions. Crowdworkers
were randomly assigned to one of two conditions: the control, where they were
asked to decide whether answers were correct or incorrect (i.e., a categorical
judgement), or the treatment, where they were shown the same question and
answers, but were instead asked to decide which of two candidate answers was
more correct (i.e., a comparative/preference-based judgement). We found that
using comparative judgement substantially improved inter-rater reliability on
both tasks. These results are in-line with well-established literature on the
benefits of comparative judgement in the field of educational assessment, as
well as with recent trends in artificial intelligence research, where
comparative judgement is becoming the preferred method for providing human
feedback on model outputs when working with non-expert crowdworkers. However,
to our knowledge, these results are novel and important in demonstrating the
beneficial effects of using the combination of comparative judgement and
crowdworkers to evaluate educational data.
Machine Learning models have many potentially beneficial applications in
education settings, but a key barrier to their development is securing enough
data to train these models. Labelling educational data has traditionally relied
on highly skilled raters using complex, multi-class rubrics, making the process
expensive and difficult to scale. An alternative, more scalable approach could
be to use non-expert crowdworkers to evaluate student work, however,
maintaining sufficiently high levels of accuracy and inter-rater reliability
when using non-expert workers is challenging. This paper reports on two
experiments investigating using non-expert crowdworkers and comparative
judgement to evaluate complex student data. Crowdworkers were hired to evaluate
student responses to open-ended reading comprehension questions. Crowdworkers
were randomly assigned to one of two conditions: the control, where they were
asked to decide whether answers were correct or incorrect (i.e., a categorical
judgement), or the treatment, where they were shown the same question and
answers, but were instead asked to decide which of two candidate answers was
more correct (i.e., a comparative/preference-based judgement). We found that
using comparative judgement substantially improved inter-rater reliability on
both tasks. These results are in-line with well-established literature on the
benefits of comparative judgement in the field of educational assessment, as
well as with recent trends in artificial intelligence research, where
comparative judgement is becoming the preferred method for providing human
feedback on model outputs when working with non-expert crowdworkers. However,
to our knowledge, these results are novel and important in demonstrating the
beneficial effects of using the combination of comparative judgement and
crowdworkers to evaluate educational data.
Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement
http://arxiv.org/abs/2305.12894v1