A Survey of the Usages of Deep Learning in Natural Language Processing <s> Question Answering <s> This paper shows how a question-answering system can be constructed using first-order logic as its language and a resolution-type theorem-prover as its deductive mechanism. A working computer program, qa3, based on these ideas is described. The performance of the program compares favorably with several other general question-answering systems. 1. QUESTION ANSWERING A question-answering system accepts information about some subject areas and answers questions by utilizing this information. The type of question-answering system considered in this paper is ideally one having the following features: 1. A language general enough to describe any reasonable question-answering subjects and express desired questions and answers. 2. The ability to search efficiently the stored information and recognize items that are relevant to a particular query. 3. The ability to derive an answer that is not stored explicitly, but that is derivable by the use of moderate effort from the stored facts. 4. Interactions between subject areas; for example, if the system has facts about Subject a and Subject b, then it should be able to answer a question that requires the use of both sets of facts. 5. Capability of allowing the user to add new facts or replace old facts conveniently. This paper argues the case for formal methods to achieve such a system and presents one particular approach in detail. A natural language facility is not one of the properties sought after or discussed (although Coles, 1968, has added to the program described here a translator from a subset of English to first-order logic). The name ‘question-answering system’ requires clarification. The system described above might be named an ‘advice taker’ or a ‘multi-purpose problem-solving system’ or ‘general problem-solving system’. McCarthy (1958) proposed using formal languages and deduction to construct such a system, and suggested allowing the user to give hints or advice on how to answer a question; he referred to the proposed system as an ‘advice taker’. Research on ‘multi-purpose’ or ‘general problem-solving’ tends to differ from question-answering as described above by placing more emphasis on solving deeper, more difficult problems and less emphasis on user interaction, formality, and efficient retrieval of relevant facts from a large data base. The situation is further confused by the use of ‘question-answering’ to refer sometimes to natural language systems, sometimes to information retrieval systems having little deductive ability, and sometimes to systems with deductive ability limited to the propositional calculus. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Question Answering <s> Answering natural language questions over a knowledge base is an important and challenging task. Most of existing systems typically rely on hand-crafted features and rules to conduct question understanding and/or answer ranking. In this paper, we introduce multi-column convolutional neural networks (MCCNNs) to understand questions from three different aspects (namely, answer path, answer context, and answer type) and learn their distributed representations. Meanwhile, we jointly learn low-dimensional embeddings of entities and relations in the knowledge base. Question-answer pairs are used to train the model to rank candidate answers. We also leverage question paraphrases to train the column networks in a multi-task learning manner. We use FREEBASE as the knowledge base and conduct extensive experiments on the WEBQUESTIONS dataset. Experimental results show that our method achieves better or comparable performance compared with baseline systems. In addition, we develop a method to compute the salience scores of question words in different column networks. The results help us intuitively understand what MCCNNs learn. <s> BIB002
Similar to summarization and information extraction, question answering (QA) gathers specific points of data, phrases, or paragraphs. However, QA is different in that it returns this information in a coherent fashion in response to a request. This problem was historically broken into the following subtasks: question classification, passage retrieval, and answer extraction  . Question classification determines the type of information requested and the format of the response that should be returned. In passage retrieval, extractive summarization is often used to retrieve, simplify, and combine information in an intelligent order to create a response. Very early approaches to answer extraction used a variety of methods such as simple parse-match methods, which parse the given requests and match these same tokens to sentences or phrases in retrieved documents  ]. Other researchers used simple first-order logic languages combined with theorem proving models such as the one proposed by BIB001 . Current models and methods resemble those used in summarization.  used a gated attention-based recurrent network to match the question with an answer-containing passage. A self-matching attention mechanism was utilized to refine the machine representation by mapping the entire passage. Finally, pointer networks were used to predict the location and boundary of an answer. These networks used attention-pooling vector representations of passages, as well as the words being analyzed, to model the critical tokens or phrases necessary for the comprehension required. This mechanism helped shift the focus from significant words throughout passages to surrounding context windows. Multicolumn convolutional neural networks (MCCNNs) were used by BIB002 to automatically analyze questions from multiple viewpoints. Parallel networks were used to extract pertinent information from input questions. Separate networks were used to find context information and relationships and to determine which forms of answers should be returned. The output of these networks was then combined and used to rank possible answers. Training on question-answer pairs, the model, without extensive tuning, learned the logical forms automatically. A recent novel approach was the use of relational networks (RNs). First proposed by , RNs are built upon an MLP architecture, where the focus is on relational reasoning, i.e. defining relationships among entities in the data. These feed-forward networks implement a similar function among all pairs of objects in order to aggregate correlations among them. For input, the RNs took final LSTM representations of document sentences. These inputs were further paired with a representation of the information request given . The RN considered all permutations to determine if there were any relationships among the sentences in a given document, or between those sentences and the presented question.