Recently, approaches for training neural language models, e.g., ELMO [1]}, GPT [2]}, BERT [3]}, RoBERTa [4]}, XLNet [5]} have led to major advancements in several NLP subfields. These methods capture dependencies between words and their compounds by pre-training neural networks on large amounts of data. Interestingly, the resulting models can be easily applied to solve different NLP applications by just fine-tuning them on the training data of the target tasks.
For example, the Transformer [6]} can be pre-trained on a large amount of data obtaining a powerful language model, which can then be specialized for solving specific NLP tasks by just adding new layers and training them on the target data.
Although the approach is simple, the procedure for fine-tuning a Transformer-based model is not completely understood, and can result in high accuracy variance, with a possible on-off behavior, e.g., models may always predict a single label. As a consequence, researchers need to invest a considerable effort in selecting suitable parameters, with no theory or a well-assessed best practice helping them. Such problem also affects QA, and, in particular, AS2 since no large and and accurate dataset has been developed for it.
