Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey <s> Representation-based Models <s> The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. ::: ::: An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible. <s> BIB001 </s> Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey <s> Representation-based Models <s> Recurrent neural networks (RNNs) are a powerful model for sequential data. End-to-end training methods such as Connectionist Temporal Classification make it possible to train RNNs for sequence labelling problems where the input-output alignment is unknown. The combination of these methods with the Long Short-term Memory RNN architecture has proved particularly fruitful, delivering state-of-the-art results in cursive handwriting recognition. However RNN performance in speech recognition has so far been disappointing, with better results returned by deep feedforward networks. This paper investigates deep recurrent neural networks, which combine the multiple levels of representation that have proved so effective in deep networks with the flexible use of long range context that empowers RNNs. When trained end-to-end with suitable regularisation, we find that deep Long Short-term Memory RNNs achieve a test set error of 17.7% on the TIMIT phoneme recognition benchmark, which to our knowledge is the best recorded score. <s> BIB002 </s> Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey <s> Representation-based Models <s> Passage-level question answer matching is a challenging task since it requires effective representations that capture the complex semantic relations between questions and answers. In this work, we propose a series of deep learning models to address passage answer selection. To match passage answers to questions accommodating their complex semantic relations, unlike most previous work that utilizes a single deep learning structure, we develop hybrid models that process the text using both convolutional and recurrent neural networks, combining the merits on extracting linguistic information from both structures. Additionally, we also develop a simple but effective attention mechanism for the purpose of constructing better answer representations according to the input question, which is imperative for better modeling long answer sequences. The results on two public benchmark datasets, InsuranceQA and TREC-QA, show that our proposed models outperform a variety of strong baselines. <s> BIB003 </s> Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey <s> Representation-based Models <s> We describe a new deep learning architecture for learning to rank question answer pairs. Our approach extends the long short-term memory (LSTM) network with holographic composition to model the relationship between question and answer representations. As opposed to the neural tensor layer that has been adopted recently, the holographic composition provides the benefits of scalable and rich representational learning approach without incurring huge parameter costs. Overall, we present Holographic Dual LSTM (HD-LSTM), a unified architecture for both deep sentence modeling and semantic matching. Essentially, our model is trained end-to-end whereby the parameters of the LSTM are optimized in a way that best explains the correlation between question and answer representations. In addition, our proposed deep learning architecture requires no extensive feature engineering. Via extensive experiments, we show that HD-LSTM outperforms many other neural architectures on two popular benchmark QA datasets. Empirical studies confirm the effectiveness of holographic composition over the neural tensor layer. <s> BIB004 </s> Text-based Question Answering from Information Retrieval and Deep Neural Network Perspectives: A Survey <s> Representation-based Models <s> Recently, pre-trained models have been the dominant paradigm in natural language processing. They achieved remarkable state-of-the-art performance across a wide range of related tasks, such as textual entailment, natural language inference, question answering, etc. BERT, proposed by Devlin this http URL., has achieved a better marked result in GLUE leaderboard with a deep transformer architecture. Despite its soaring popularity, however, BERT has not yet been applied to answer selection. This task is different from others with a few nuances: first, modeling the relevance and correctness of candidates matters compared to semantic relatedness and syntactic structure; second, the length of an answer may be different from other candidates and questions. In this paper. we are the first to explore the performance of fine-tuning BERT for answer selection. We achieved STOA results across five popular datasets, demonstrating the success of pre-trained models in this task. <s> BIB005
is a candidate answer for question q i , and label y ij shows whether a ij is a correct answer for q i or not. For each answer, a related question is generated and then the semantic similarity of generated question and the given question is captured by using dot product. This similarity is used for predicting whether the candidate answer is a correct answer for given question or not. The probability of the answer being correct is formulated as: where q = M a is the generated question. The model is trained by minimizing the cross entropy of all labelled data QA pairs as: where θ 2 F is the Frobenius norm of θ. Each sentence is modeled by bag of word and bigram approaches. In bag of words model, a sentence is represented by averaging embeddings of all the words (except stop words) within it. The bigram model has the ability to capture features of bigrams independent of their positions in the sentence. As architecture of bigram model is shown in Figure 2 , one convolutional layer and one pooling layer is used for modeling the sentence in bigram model. Every bigram is projected into a feature value c i , which is computed as: where s is the vector representation of the sentence. All bigram features are combined in average pooling layer and finally a full-sentence representation with the same dimensionality as the initial word embeddings is produced by the following equation: Severyn and Moschitti (2015) proposed a framework for answer sentence selection. They divided their task into two main subtasks: (1) mapping the original space of words to a feature space encoding, and (2) learning a similarity function between pairs of objects. They used a Convolutional Neural Network (CNN) architecture for learning to map input text, either query or document, to a vector space model. For the second part, they used the idea of noisy channel approach to find a transformation of document to be as close as possible to the query: To this end, they used a neural network architecture to train the similarity matrix M . According to Figure  3 , the vector representation of query and document that are derived from the first CNN model are jointly fed to the second CNN to train and build the similarity matrix. Wang and Nyberg (2015) used a multilayer stacked Bidirectional Long Short-term Memory (BiLSTM) for answer sentence selection task. As represented in Figure 4 , a sequence of word2vec representation of question and answer sentence terms are fed to this model. Symbol, < S >, is placed between question q and answer a for distinguishing the question and answer. Among different Recurrent Neural Network (RNN) architectures, stacked BiLSTM is chosen as first bidirectional RNN extracts the contextual information of question and answer pair from both directions in other words it uses the future information, second stacked BiLSTM provides better results due to its ability in extracting higher levels of abstraction, and third LSTM is a more complicated RNN block which mitigates the gradient vanishing problem of standard RNNs. The final output of each time step indicates whether the given answer is a correct answer for the question or not. In this model the stacked BiLSTM relevance model is combined by Gradient Boosted Regression Tree (GBDT) method for exact matching the proper nouns and cardinal numbers in question and answer sentences. BIB004 proposed Holographic-dual LSTM (HD-LSTM), a binary classifier model for QA task. As is shown in Figure 5 , HD-LSTM consists four major parts. In representation layer, two multi-layered LSTMs denoted as Q-LSTM and A-LSTM are used for learning the representations of the question and answer. A holographic composition is used for measuring the similarity of the outputs of Q-LSTM and A-LSTM. Finally, a fully connected hidden Similarity of question and answer is modeled by the following equation:. where F is Fast Fourier transform, q is question, a is answer, and d is dimension of embeddings. In the above equation, question and answer embeddings must have the same dimension. 3. Holographic Hidden Layer: This is a fully connected dense layer. Input and output of this layer are [[q a], Sim(q, a), X f eat ] and h out , respectively. (X f eat ) is word overlap feature, and Sim(q, a) is bilinear similarity function between q and a which is defined as: where M ∈ R n×n is a similarity matrix between q ∈ R n and a ∈ R n . Concatenation of Sim(q, a) with [q a] makes the model perform worse. So, in order to mitigate this weakness (X f eat ) is concatenated to make model work better.  where W f and b f are network parameters. BIB003 proposed a basic model called QA-LSTM, shown in Figure 6 , for sentence matching. According to this figure, in the basic model, word embeddings of question and answer sentences are fed into a BiLSTM network. A fixed size representation is obtained for each sentence in three different ways: (1) concatenating the last output of both directions, (2) average pooling and max pooling over all the outputs of the BiLSTM, and finally, (3) using the cosine similarity, semantic matching between question and answer sentences is scored. LSTM is a powerful architecture in capturing long range dependencies, but it suffers from not paying attention to local n-grams. Although convolutional structures pay more attention to local n-grams, they do not consider long range dependencies. Therefore each of the CNN and RNN blocks has their own pros and cons. Three different variants of the basic QA-LSTM are proposed in this work which one of them belongs to the hybrid models and is described in section 6.3. In the following, two other variants of QA-LSTM, belonging to the representation-based models, are described. 1. Convolutional pooling LSTM: As is shown in Figure 7 , pooling layer is replaced with a convolutional layer for capturing richer local information and on top of this layer an output layer is placed for generating representation of the input sentence. Representation of the input sentence is generated by the following equations: where Z ∈ R k|h|×L and m-th column is generated by concatenation of the k hidden vectors of BiLSTM centralized in the m-th token of the sequence, L is length of the sequence, and W cp is the network parameter. Figure  8 . In this model, word embeddings are first fed to a CNN for retrieving the local n-gram interactions at the lower level. Output of the convolution is then fed into the BiLSTM network for capturing long-range dependencies. Max pooling is used over the output of the BiLSTM for producing the sentence representation. Output of the convolution layer, named X, is obtained by the following equation: where D ∈ R kE×L is input of this model and column l of D is concatenation of k word vectors of size E centered at the l-th word. ABCNN models belong to the hybrid category and are described in section 6.3. In the following, architecture of BCNN is described. Basic CNN (BCNN): Architecture of this model is shown in Figure 9 . This model is based on the Siamese architecture  . The model provides representation of each sentence using convolutional, w − ap pooling and all − ap pooling layers, and then compares these two representations with logistic regression. Different layers in BCNN are as follows: 1. Input layer: Each sentence is passed to the model with a d 0 × s matrix, where d 0 is the dimension of word2vec BIB001 embedding of each word and s is maximum length of the two sentences (the shorter sentence is padded to the larger sentence length). 2. Convolution layer: Embedding of words within a sentence with window size of w are concatenated and represented as c i ∈ R w.d0 where 0 < i < s+w (s is length of the sentence). Then each c i is converted to p i by the following equation: let W ∈ R d1×w.d0 be the convolution weights, and b ∈ R d1 be the bias. 3. Average pooling layer: This model utilizes two types of average pooling, namely all − ap and w − ap, for extracting the robust features from convolution. All − ap pooling is used in the last convolution layer and calculates average of each column. W − ap pooling is used in the middle convolution layers and calculates average of each w consecutive columns. 4. Output layer: In the output layer logistic regression is applied to final representations in order to classify the question and answer sentences as related or not. He and Lin (2016) proposed a model for QA task which consists of four major components. Architecture of the model is shown in Figure 10 . Different components of this model are described below: Various length of answer sentences results in variable size for QA matrix. To fix this problem, value shared weights method is used. In value shared weights method, each node is weighted based on its value where value of a node represents the similarity between two words. Input to the hidden layer for each question term is defined as follows: where j is the index of the question term, w k is the model parameter, and x jk is the sum of all matching signals within the range k (the range of possible matching signals is divided to k equal bins, and each matching score is assigned to one bin). 3. Question attention network: An attention layer with question word embedding weights is applied to hidden states h j . Finally match score is computed by the following equation:  where v is model parameter and dot product of the question word embedding and v are fed to the softMax function. In aNNM-2, more than one value-shared weights are used for each question answer matching vector then in the first hidden layer there are multiple intermediate nodes. As architecture of aNMM-2 is shown in the Figure 11, final output of the model y is defined as: where T is the number of nodes in hidden layer 1, r t is the model parameter from hidden layer 1 to hidden layer 2, and τ (v, q j ) is calculated as follows: Wan et al. (2016b) proposed a recursive semantic matching model called Match-SRNN. According to Figure 12 , which shows the architecture of Match-SRNN, Match-SRNN works in three major steps: In the first step word-level interactions are modeled. In the second step a special case of interaction between two prefixes of two different sentence (S1[1 : i] and S2[1 : j] ) is modeled as a function of the interaction between S1[1 : i − 1] and S2[1 : j], S1[1 : i] and S2[1 : j − 1], S1[1 : i − 1] and S2[1 : j − 1], and interaction between words w i and v j . Then the equation for this special interaction is: where h ij is the interaction between S1[1 : i] and S2[1 : j]. This recursive way of modeling the interaction helps to capture the long-term dependencies between two sentences. In the third step a linear function is used for measuring the matching score of two given sentences. More details about these steps are below. 1. Neural tensor network: The interaction between two words w i and v j is captured by a neural tensor network according to the following equation: where s ij is a vector representation of the similarity between w i and v j words, T i is one slice of the tensor parameters, W and b are parameters and F (Z) = max(0, Z). 2. Spatial RNN: In this layer, GRU is used as an RNN because of its easy implementation for implementing a Spatial-GRU which models the h ij . Figure 13 shows architecture of the 1D-GRU and Spatial-GRU which is used in this work. According to the right part in Figure 13 Spatial-GRU has four updating gates, and three reset gates. Function f in the Spatial-GRU is computed as follow: 3. Linear Scoring Function: Final matching score of two given sentences is computed by the equation: where h mn is the global interaction between two sentences, and W (s) and b (s) are network parameters. Devlin et al. (2019) proposed Bidirectional Encoder Representations from Transformers (BERT) model which is a language modeling neural network. BERT has a multi-layered bidirectional architecture, in which each layer is a transformer encoder. Transformer was proposed by Vaswani et al. (2017) and has an encoder-decoder architecture. The same decoder segment of transformer model is used in BERT. BERT is used in a wide range of NLP tasks, including QA, natural language inference, and text classification. BERT is pretrained on large corpora by two different approaches, namely masked language model and next sentence prediction, and then fine-tuned on each specific task based on the application. Architecture of BERT including both pre-training and fine-tuning steps is shown in Figure 14 . Input of BERT is a sequence of input representation of words. Input representation of each word is built by summing its token embeddings, segment embeddings, and position embeddings as shown in Figure 15 . In the QA domain, a (CLS) token is used in the first position of the sequence, then the question and the candidate answer followed by a (SEP ) token are placed in the sequence, respectively. Output of BERT is encoded representation for each token. BIB005 proposed BERTSel which fine-tunes BERT model for answer selection task as is shown in Figure 16 . Each question (q) with a correct answer (p) and a wrong answer (n) are fed to this model (q, p, n). In the finetuning step, the question once with the correct answer (q, p) and once with the wrong answer (q, n) is passed to BERT. In this model, encoded representation of [CLS] token is passed to a fully-connected layer followed by a sigmoid function for predicting the matching score of given question and answer pair. of these two sentences with different similarity metrics. Then a k-max pooling layer and a multi-layerd LSTM are used for modeling the matching score of two given sentences. Architecture of MV-LSTM is explained in more details in following three steps: 1. Positional Sentence Representation: Positional sentence representation or representation of sentence at one position is obtained by BiLSTM . BiL-STM is used for capturing the long and short-term dependencies in one sentence. An LSTM layer similar to the implementation used in BIB002 is used here. Given a sentence S = (x 0 , x 1 , ..., x T ), LSTM represents position t of the sentence h t as follows: Utilizing the BiLSTM layer, two different representations − → h t and ← − h t are generated for each position and final representation of each position is considered as the concatenation of these two representations [