CONDA QA: A Contrastive Reading Comprehension Dataset for
Reasoning about Negation
Abhilasha Ravichander
Carnegie Mellon Univer
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation
The full power of human language-based communication cannot be realized
without negation. All human languages have some form of negation. Despite this,
negation remains a challenging phenomenon for current natural language
understanding systems. To facilitate the future development of models that can
process negation effectively, we present CONDAQA, the first English reading
comprehension dataset which requires reasoning about the implications of
negated statements in paragraphs. We collect paragraphs with diverse negation
cues, then have crowdworkers ask questions about the implications of the
negated statement in the passage. We also have workers make three kinds of
edits to the passage -- paraphrasing the negated statement, changing the scope
of the negation, and reversing the negation -- resulting in clusters of
question-answer pairs that are difficult for models to answer with spurious
shortcuts. CONDAQA features 14,182 question-answer pairs with over 200 unique
negation cues and is challenging for current state-of-the-art models. The best
performing model on CONDAQA (UnifiedQA-v2-3b) achieves only 42% on our
consistency metric, well below human performance which is 81%. We release our
dataset, along with fully-finetuned, few-shot, and zero-shot evaluations, to
facilitate the development of future NLP methods that work on negated language.
Abhilasha Ravichander Matt Gardner Ana Marasović