Neural models have significantly contributed to both areas with new techniques, e.g., [1]}, [2]}, [3]}. In particular, recent approaches to neural language models, e.g., ELMO [4]},
BERT [5]}, RoBERTa [6]}, XLNet [7]} have led to major advancements in several NLP subfields.
These methods capture dependencies between words and their compounds by pre-training neural networks on large amounts of data. Interestingly, the resulting models can be easily applied to different tasks by fine-tuning them on the target training data.
The impact of such methods on AS2, also thanks to transfer learning, is impressive. For example, [8]} exceeded the state of the art by 50% (relative error reduction) on WikiQA [9]} and TREC-QA [10]} datasets.
However, the proposed Transformer methods only focus on the similarity between the question and the candidate sentence pairs, without taking any additional information into account.
