
We improve stability of Transformer models by adding an intermediate fine-tuning step, which aims at specializing them to the target task (AS2), i.e., this step transfers a pre-trained language model to a model for the target task.

We show that the transferred model can be effectively adapted to the target domain with a subsequent fine-tuning step, even when using target data of small size.

Our Transfer and Adapt (TandA) approach makes fine-tuning: (i) easier and more stable, without the need of cherry picking parameters; and (ii) robust to noise, i.e., noisy data from the target domain can be utilized to train an accurate model.

We built ASNQ, a dataset for AS2, by transforming the recently released Natural Questions (NQ) corpus [1]} from MR to AS2 task. This was essential as our transfer step requires a large and accurate dataset. ASNQ is an important contribution of our work to the research community. The ASNQ dataset and trained models can be accessed at https://github.com/alexa/wqa_tanda

Finally, the generality of our approach and empirical investigation suggest that our TandA findings also apply to other NLP tasks, especially, textual inference, although empirical analysis is essential to confirm these claims.

