We implement RoBERTa in FAIRSEQ [1]}. RoBERTa is optimized with Adam [2]} with the following parameters: \(\beta _{1} = 0.9, \beta _{2} = 0.98, \epsilon = 1e-6\)  and \(L_{2}\)  weight decay of 0.01. The learning rate is warmed up over the first 800 steps to a peak value of 1e-5, and then polynomially decayed. The clipping threshold of gradients is 0.1. RoBERTa is fine-tuned with a dropout of 0.1 on all layers and attention weights. It is fine-tuned for S=8,000 updates, with mini-batches containing B=8 sequences of maximum length T=512 tokens.
