Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop
We present our experience as annotators in the creation of high-quality,
adversarial machine-reading-comprehension data for extractive QA for Task 1 of
the First Workshop on Dynamic Adversarial Data Collection (DADC). DADC is an
emergent data collection paradigm with both models and humans in the loop. We
set up a quasi-experimental annotation design and perform quantitative analyses
across groups with different numbers of annotators focusing on successful
adversarial attacks, cost analysis, and annotator confidence correlation. We
further perform a qualitative analysis of our perceived difficulty of the task
given the different topics of the passages in our dataset and conclude with
recommendations and suggestions that might be of value to people working on
future DADC tasks and related annotation interfaces.