Open-domain question answering (QA) is an important problem in AI and NLP that is emerging as a bellwether for progress on the generalizability of AI methods and techniques. Much of the progress in open-domain QA systems has been realized through advances in information retrieval methods and corpus construction. In this paper, we focus on the recently introduced ARC Challenge dataset, which contains 2,590 multiple choice questions authored for grade-school science exams. These questions are selected to be the most challenging for current QA systems, and current state of the art performance is only slightly better than random chance. We present a system that reformulates a given question into queries that are used to retrieve supporting text from a large corpus of science-related text. Our rewriter is able to incorporate background knowledge from ConceptNet and -- in tandem with a generic textual entailment system trained on SciTail that identifies support in the retrieved results -- outperforms several strong baselines on the end-to-end QA task despite only being trained to identify essential terms in the original source question. We use a generalizable decision methodology over the retrieved evidence and answer candidates to select the best answer. By combining query reformulation, background knowledge, and textual entailment our system is able to outperform several strong baselines on the ARC dataset. The recently released AI2 Reasoning Challenge (ARC) and accompanying ARC Corpus is an ambitious test for AI systems that perform open-domain question answering (QA). This dataset consists of 2590 multiple choice questions authored for grade-school science exams; the questions are partitioned into an Easy set and a Challenge set. The Challenge set comprises questions that cannot be answered correctly by either a Pointwise Mutual Information (PMI-based) solver, or by an Information Retrieval (IR-based) solver. also note that the simple information retrieval (IR) methodology (Elasticsearch) that they use is a key weakness of current systems, and conjecture that 95% of the questions can be answered using ARC corpus sentences.ARC has proved to be a difficult dataset to perform well on, particularly its Challenge partition: existing systems like KG 2 achieve 31.70% accuracy on the test partition. Older models such as DecompAttn BID27 and BiDAF that have shown good performance on other datasets -e.g. SQuAD BID29 ] -perform only 1-2% above random chance. 1 The seeming intractability of the ARC Challenge dataset has only very recently shown signs of yielding, with the newest techniques attaining an accuracy of 42.32% on the Challenge set BID35 . 2 An important avenue of attack on ARC was identified in Boratko et al. [2018a,b] , which examined the knowledge and reasoning requirements for answering questions in the ARC dataset. The authors note that "simple reformulations to the query can greatly increase the quality of the retrieved sentences". They quantitatively measure the effectiveness of such an approach by demonstrating a 42% increase in score on ARC-Easy using a pre-trained version of the DrQA model BID7 . Another recent tack that many top-performing systems for ARC have taken is the use of natural language inference (NLI) models to answer the questions . The NLI task -also sometimes known as recognizing textual entailment -is to determine whether a given natural language hypothesis h can be inferred from a natural language premise p. The NLI problem is often cast as a classification problem: given a hypothesis and premise, classify their relationship as either entailment, contradiction, or neutral. NLI models have improved state of the art performance on a number of important NLP tasks BID27 and have gained recent popularity due to the release of large datasets BID4 BID46 BID43 . In addition to the NLI models, other techniques applied to ARC include using pre-trained graph embeddings to capture commonsense relations between concepts BID51 ; as well as the current state-of-theart approach that recasts multiple choice question answering as a reading comprehension problem that can also be used to fine-tune a pre-trained language model BID35 .ARC Challenge represents a unique obstacle in the open domain QA world, as the questions are specifically selected to not be answerable by merely using basic techniques augmented with a high quality corpus. Our approach combines current best practices: it retrieves highly salient evidence, and then judges this evidence using a general NLI model. While other recent systems for ARC have taken a similar approach BID26 BID25 , our extensive analysis of both the rewriter module as well as our decision rules sheds new light on this unique dataset.In order to overcome some of the limitations of existing retrieval-based systems on ARC and other similar corpora, we present an approach that uses the original question to produce a set of reformulations. These reformulations are then used to retrieve additional supporting text which can then be used to arrive at the correct answer. We couple this with a textual entailment model and a robust decision rule to achieve good performance on the ARC dataset. We discuss important lessons learned in the construction of this system, and key issues that need to be addressed in order to move forward on the ARC dataset. Of the systems above ours on the leaderboard, only BID26 report their accuracy on both the dev set (43.29%) and the test set (36.36%). We suffer a similar loss in performance from 36.37% to 33.20%, demonstrating the risk of overfitting to a (relatively small) development set in the multiplechoice setting even when a model has few learnable parameters. As in this paper, BID26 pursue the approach suggested by Boratko et al. [2018a,b] in learning how to transform a naturallanguage question into a query for which an IR system can return a higher-quality selection of results. Both of these systems use entailment models similar to our match-LSTM BID41 model, but also incorporate additional co-attention between questions, candidate answers, and the retrieved evidence. BID35 present an an encouraging result for combating the IR bottleneck in opendomain QA. By concatenating the top-50 results of a single (joint) query and feeding the result into a neural reader optimized by several lightly-supervised 'reading strategies', they achieve an accuracy of 37.4% on the test set even without optimizing for single-answer selection. Integrating this approach with our query rewriting module is left for future work. In this paper, we present a system that answers science exam questions by retrieving supporting evidence from a large, noisy corpus on the basis of keywords extracted from the original query. By combining query rewriting, background knowledge, and textual entailment, our system is able to outperform several strong baselines on the ARC dataset. Our rewriter is able to incorporate background knowledge from ConceptNet and -in tandem with a generic entailment model trained on SciTail -achieves near state of the art performance on the end-to-end QA task despite only being trained to identify essential terms in the original source question.There are a number of key takeaways from our work: first, researchers should be aware of the impact that Elasticsearch (or a similar tool) can have on the performance of their models. Answer candidates should not be discarded based on the relevance score of their top result; while (correct) answers are likely critical to retrieving relevant results, the original AI2 Rule is too aggressive in pruning candidates. Using an entailment model that is capable of leveraging background knowledge in a more principled way would likely help in filtering unproductive search results. Second, our results corroborate those of BID26 and show that tuning to the dev partition of the Challenge set (299 questions) is extremely sensitive. Though we are unable to speculate on whether this is an artifact of the dataset or a more fundamental concern in multiple-choice QA, it is an important consideration for generating significant and reproducible improvements on the ARC dataset.
We explore how using background knowledge with query reformulation can help retrieve better supporting evidence when answering multiple-choice science questions.
{"src": "scitldr-AIC_train", "paper_id": "HJxYZ-5paX"}