Reading comprehension tasks test the ability of models to process long-term context and remember salient information. Recent work has shown that relatively simple neural methods such as the Attention Sum-Reader can perform well on these tasks; however, these systems still significantly trail human performance. Analysis suggests that many of the remaining hard instances are related to the inability to track entity-references throughout documents. This work focuses on these hard entity tracking cases with two extensions: (1) additional entity features, and (2) training with a multi-task tracking objective. We show that these simple modifications improve performance both independently and in combination, and we outperform the previous state of the art on the LAMBADA dataset, particularly on difficult entity examples.
A number of datasets in this style exist with different focus. Here we considered the LAMBADA dataset and the named entity portion of the Children's Book Test dataset (CBT-NE). LAMBADA uses novels where examples consist of 4-5 sentences and the last word to be predicted is masked, @math . The dataset is constructed carefully to focus on examples where humans needed the context to predict the masked word. CBT-NE examples, on the other hand, include 21 sentences where the masked word is a named entity extracted from the last sentence, with @math , and is constructed in a more automated way. We show an example from LAMBADA in Figure . In CBT, as well as the similar CNN Daily Mail dataset @cite_2 , the answer @math is always contained in @math whereas in LAMBADA it may not be. showed, however, that training only on examples where @math in @math leads to improved overall performance, and we adopt this approach as well.
@cite_2
2949615363
Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.