Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey <s> 1. <s> This paper presents a syntax-driven approach to question answering, specifically the answer-sentence selection problem for short-answer questions. Rather than using syntactic features to augment existing statistical classifiers (as in previous work), we build on the idea that questions and their (correct) answers relate to each other via loose but predictable syntactic transformations. We propose a probabilistic quasi-synchronous grammar, inspired by one proposed for machine translation (D. Smith and Eisner, 2006), and parameterized by mixtures of a robust nonlexical syntax/alignment model with a(n optional) lexical-semantics-driven log-linear model. Our model learns soft alignments as a hidden variable in discriminative training. Experimentalresultsusing theTRECdataset are shown to significantly outperform strong state-of-the-art baselines. <s> BIB001 </s> Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey <s> 1. <s> We describe the WIKIQA dataset, a new publicly available set of question and sentence pairs, collected and annotated for research on open-domain question answering. Most previous work on answer sentence selection focuses on a dataset created using the TREC-QA data, which includes editor-generated questions and candidate answer sentences selected by matching content words in the question. WIKIQA is constructed using a more natural process and is more than an order of magnitude larger than the previous dataset. In addition, the WIKIQA dataset also includes questions for which there are no correct sentences, enabling researchers to work on answer triggering, a critical component in any QA system. We compare several systems on the task of answer sentence selection on both datasets and also describe the performance of a system on the problem of answer triggering using the WIKIQA dataset. <s> BIB002 </s> Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey <s> 1. <s> We apply a general deep learning framework to address the non-factoid question answering task. Our approach does not rely on any linguistic tools and can be applied to different languages or domains. Various architectures are presented and compared. We create and release a QA corpus and setup a new QA task in the insurance domain. Experimental results demonstrate superior performance compared to the baseline methods and various technologies give further improvements. For this highly challenging task, the top-1 accuracy can reach up to 65.3% on a test set, which indicates a great potential for practical use. <s> BIB003 </s> Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey <s> 1. <s> We introduce the MovieQA dataset which aims to evaluate automatic story comprehension from both video and text. The dataset consists of 14,944 questions about 408 movies with high semantic diversity. The questions range from simpler "Who" did "What" to "Whom", to "Why" and "How" certain events occurred. Each question comes with a set of five possible answers; a correct one and four deceiving answers provided by human annotators. Our dataset is unique in that it contains multiple sources of information -- video clips, plots, subtitles, scripts, and DVS. We analyze our data through various statistics and methods. We further extend existing QA techniques to show that question-answering with such open-ended semantics is hard. We make this data set public along with an evaluation benchmark to encourage inspiring work in this challenging domain. <s> BIB004 </s> Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey <s> 1. <s> For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation benchmark (GLUE), a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks. GLUE is model-agnostic, but it incentivizes sharing knowledge across tasks because certain tasks have very limited training data. We further provide a hand-crafted diagnostic test suite that enables detailed linguistic analysis of NLU models. We evaluate baselines based on current methods for multi-task and transfer learning and find that they do not immediately give substantial improvements over the aggregate performance of training a separate model per task, indicating room for improvement in developing general and robust NLU systems. <s> BIB005
WikiQA is an open domain QA dataset BIB002 . This dataset is collected from Bing query logs. Question-like queries which are issued by at least 5 different users and have clicks to Wikipedia pages are selected as questions in this dataset and the sentences of the summary section of the corresponding Wikipedia page are considered as candidate answers to the related question. The candidate answers are labeled as correct or incorrect with crowdsourcing and then the correct answer is selected. This dataset consists 3047 questions and 1473 answers, more statistics about this dataset is presented in Table 1 . A noted feature of WikiQA is that not all the questions in this dataset have the correct answer which makes it possible to use this dataset in answer triggering component. Answer triggering component's task is finding whether the question has answer or not. BIB001 . Questions of TREC 8-12 are used as training dataset and questions of TREC 13 are used as development and test dataset. Statistics of TREC-QA dataset is presented in Table 2 . TREC-QA contains two training datasets, TRAIN and TRAIN-ALL. TRAIN dataset includes first 94 questions of TREC 8-12 and its candidate answers are judged manually, while in TRAIN-ALL dataset, correct answers are recognized by matching the answers with predefined patterns of the answer regular expression. 3. MovieQA is gathered from diverse data sources BIB004 which is a unique feature of this dataset. It contains 14944 questions where each question is associated with five answers, including one correct answer and four deceiving answers. Statistics of this dataset is shown in Table 3 . 4. InsuranceQA is a close domain dataset for QA in insurance domain BIB003 . Question/answer pairs of this dataset are collected from internet. This dataset includes Train, Development, Test1, and Test2 parts. More detailed statistics about InsuranceQA is presented in Table 4 . 5. Yahoo! Dataset is collected from Yahoo! Answers QA system. Yahoo! includes 142,627 question/answer pairs. But in the literature BIB005 ,a) a subset of this dataset is selected as positive pair and for each question in this subset, four other negative pairs are constructed. The (question, answer) pairs which the question and its answer length is between 5 to 50 words are selected as positive pairs (including 60564 pairs). Four negative answers for each question are selected by querying the whole answers set by the correct answer and selecting the four answers randomly among 1000 top retrieved answers.