Think you have Solved Question Answering?
Try ARC, the AI2 Reasoning Challenge
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar K
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
We present a new question set, text corpus, and baselines assembled to
encourage AI research in advanced question answering. Together, these
constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful
knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC
question set is partitioned into a Challenge Set and an Easy Set, where the
Challenge Set contains only questions answered incorrectly by both a
retrieval-based algorithm and a word co-occurence algorithm. The dataset
contains only natural, grade-school science questions (authored for human
tests), and is the largest public-domain set of this kind (7,787 questions). We
test several baselines on the Challenge Set, including leading neural models
from the SQuAD and SNLI tasks, and find that none are able to significantly
outperform a random baseline, reflecting the difficult nature of this task. We
are also releasing the ARC Corpus, a corpus of 14M science sentences relevant
to the task, and implementations of the three neural baseline models tested.
Can your model perform better? We pose ARC as a challenge to the community.
Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot Ashish Sabharwal Carissa Schoenick Oyvind Tafjord