We implemented our methods with RoBERTa pre-trained models, using the shared checkpoint [1]}. We fine-tuned the checkpoint on our data by using (i) the Adam optimizer set with the warmup linear scheduler and a learning rate peak of \(1e-6\) ; (ii) the binary cross-entropy loss; (iii) a batch size of 64 examples on a single GPU to train on WikiQA and SQuAD; and (iii) a batch size of 512 examples distributed on 8 GPUs to train on the ASNQ corpus (which is much larger).
We used the official dev. set to derive the results, thus we set the parameters, i.e., learning rate, scheduler, and batch size, on a small portion of the training set (as our dev. set).
We train and test our models on SQuAD and WikiQA four times and take the average results to account for their variability.
Finally, we also used the models generated with TANDA (transfer and adapt) approach [2]} for WikiQA. The authors apply a first fine-tuning on ASNQ and then a second fine-tuning on the target data. TANDA is the current state of the art, 7-10 points better than any other approach on WikiQA.
