We propose TandA, an effective technique for fine-tuning pre-trained Transformer models for natural language tasks. Specifically, we firsttransfera pre-trained model into a model for a general task by fine-tuning it with a large and high-quality dataset. We then perform a second fine-tuning step toadaptthe transferred model to the target domain. We demonstrate the benefits of our approach for answer sentence selection, which is a well-known inference task in Question Answering. We built a large scale dataset to enable the transfer step, exploiting the Natural Questions dataset.
Our approach establishes the state of the art on two well-known benchmarks, WikiQA and TREC-QA, achieving MAP scores of 92% and 94.3%, respectively, which largely outperform the previous highest scores of 83.4% and 87.5%, obtained in very recent work.
We empirically show that TandA generates more stable and robust models reducing the effort required for selecting optimal hyper-parameters.
Additionally, we show that the transfer step of TandA makes the adaptation step more robust to noise.
This enables a more effective use of noisy datasets for fine-tuning. Finally, we also confirm the positive impact of TandA in an industrial setting, using domain specific datasets subject to different types of noise.