
QA-LSTM [1]} is based on bidirectional LSTMs and attention. Both query and document are encoded using a shared bidirectional many-to-many LSTM and a pooling operation (maximum or average pooling) to the LSTM outputs. Attention scores are computed using the hidden LSTM states of the document and the pooled query representation. The resulting vectors are then compared using cosine similarity after applying dropout. We set the batch size to 32 and the number of LSTM hidden units to 256. We feed 300-dimensional pre-trained GloVe [2]} embeddings to the shared LSTM and use a dropout rate of \(0.5\) .

K-NRM [3]} is a neural ranking model that works via kernel pooling. Starting from pre-trained word embeddings, it builds a translation matrix, where each row contains the cosine similarities of a query word to all document words. Each row is then fed into \(K\)  kernel functions, and the results are pooled by summation. Finally, a single transformation with tanh activation is applied to output a score. The model is trained with a pairwise ranking loss and uses RBF kernels. We use 300-dimensional pre-trained GloVe embeddings to build the translation matrix. The hyperparameters are adopted from [3]}: We set \(K = 11\)  and use one kernel for exact matches, i.e. \(\mu _0 = 1\)  and \(\sigma _0 = 10^{-3}\) . The remaining kernels are spaced evenly in \([-1, 1]\)  with \(\mu _1 = 0.9\) , \(\mu _2 = 0.7\) , ..., \(\mu _{10} = -0.9\)  and \(\sigma _1 = ... = \sigma _{10} = 0.1\) . We use the Adam optimizer with a leaning rate of \(0.001\)  and \(\epsilon = 10^{-5}\)  and a batch size of 16.

Dynamic Memory Network [5]}, [6]} serves (in a slightly modified fashion) as the aggregation part of our model, which transforms sentence-level BERT outputs into a relevance score. We also train this model using pre-trained 300-dimensional word vectors in order to analyze if and how much BERT representations improve the performance. For these experiments we use the same DMN hyperparameters as in our experiments with BERT-DMN to make the results more comparable.

