Common-sense or background knowledge is required to understand natural language, but in most neural natural language understanding (NLU) systems, the requisite background knowledge is indirectly acquired from static corpora. We develop a new reading architecture for the dynamic integration of explicit background knowledge in NLU models. A new task-agnostic reading module provides refined word representations to a task-specific NLU architecture by processing background knowledge in the form of free-text statements, together with the task-specific inputs. Strong performance on the tasks of document question answering (DQA) and recognizing textual entailment (RTE) demonstrate the effectiveness and flexibility of our approach. Analysis shows that our models learn to exploit knowledge selectively and in a semantically appropriate way. Understanding natural language depends crucially on common-sense or background knowledge, for example, knowledge about what concepts are expressed by the words being read (lexical knowledge), and what relations hold between these concepts (relational knowledge). As a simple illustration, if an agent needs to understand that the statement "King Farouk signed his abdication" is entailed by "King Farouk was exiled to France in 1952, after signing his resignation", it must know (among other things) that abdication means resignation of a king.In most neural natural language understanding (NLU) systems, the requisite background knowledge is implicitly encoded in the models' parameters. That is, what background knowledge is present has been learned from task supervision and also by pre-training word embeddings (where distributional information reliably reflects certain kinds of useful background knowledge, such as semantic relatedness). However, acquisition of background knowledge from static training corpora is limiting for two reasons. First, we cannot expect that all background knowledge that could be important for solving an NLU task can be extracted from a limited amount of training data. Second, as the world changes, the facts that may influence how a text is understood will likewise change. In short: building suitably large corpora to capture all relevant information, and keeping the corpus and derived models up to date with changes to the world would be impractical.In this paper, we develop a new architecture for dynamically incorporating external background knowledge in NLU models. Rather than relying only on static knowledge implicitly present in the training data, supplementary knowledge is retrieved from a knowledge base to assist with understanding text inputs. Since NLU systems must necessarily read and understand text inputs, our approach incorporates background knowledge by repurposing this reading machinery-that is, we read the text being understood together with supplementary natural language statements that assert facts (assertions) which are relevant to understanding the content ( §2).Our knowledge-augmented NLU systems operate in a series of phases. First , given the text input that the system must understand, which we call the context, a set of relevant supporting assertions is retrieved. While learning to retrieve relevant information for solving NLU tasks is an important question BID21 Narasimhan et al., 2016, inter alia) , in this work, we focus on learning how to incorporate retrieved information, and use simple heuristic retrieval methods to identify plausibly relevant background from an external knowledge base. Once the supplementary texts have been retrieved, we use a word embedding refinement strategy that incrementally reads the context and retrieved assertions starting with context-independent word embeddings and building successively refined embeddings of the words that ultimately reflect both the relevant supporting assertions and the input context ( §3). These contextually refined word embeddings, which serve as dynamic memory to store newly incorporated knowledge, are used in any task-specific reading architecture. The overall architecture is illustrated in FIG0 . Although we are incorporating a new kind of information into the NLU pipeline, a strength of this approach is that the architecture of the reading module is independent of the final NLU task-the only requirement is that the final architecture use word embeddings.We carry out experiments on several different datasets on the tasks of document question answering (DQA) and recognizing textual entailment (RTE) evaluating the impact of our proposed solution with both basic task architectures and a sophisticated task architecture for RTE ( §4). We find that our embedding refinement strategy is quite effective ( §5). On four standard benchmarks, we show that refinement helps-even refining the embeddings just using the context (and no additional background information) can improve performance significantly, and adding background knowledge helps further. Our results are very competitive, setting a new state-of-the-art on the recent TriviaQA benchmarks which is remarkable considering the simplicity of the chosen task-specific architecture. Finally, we provide a detailed analysis of how knowledge is being used by an RTE system ( §6), including experiments showing that our system is capable of making appropriate counterfactual inferences when provided with "false knowledge". We have presented a novel task-agnostic reading architecture that allows for the dynamic integration of background knowledge into neural NLU models. Our solution, which is based on the incremental refinement of word representations by reading supplementary inputs, is flexible and be used with virtually any existing NLU architecture that rely on word embeddings as input. Our results show that embedding refinement using both the system's text inputs, as well as supplementary texts encoding background knowledge can yield large improvements. In particular, we have shown that relatively simple task architectures (e.g., based on simple BiLSTM readers) can become competitive with state-of-the-art, task-specific architectures when augmented with our reading architecture.
In this paper we present a task-agnostic reading architecture for the dynamic integration of explicit background knowledge in neural NLU models. 
{"src": "scitldr-AIC_train", "paper_id": "B1twdMCab"}