BART is also implemented in FAIRSEQ [1]}. It is optimized with Adam [2]} with the following parameters: \(\beta _{1} = 0.9, \beta _{2} = 0.999, \epsilon = 1e-8\)  and \(L_{2}\)  weight decay of 0.01. The learning rate is warmed up over the first 500 steps to a peak value of 3e-5, and then polynomially decayed. The clipping threshold of gradients is 0.1. BART is fine-tuned with a dropout of 0.1 on all layers and attention weights. It is fine-tuned for S=1,200 updates, with mini-batches containing B=32 sequences of maximum length T=512 tokens. Note that we keep the same experiment settings in the following experiments. We train the model on 4 \(\times \)  11GB GeForce RTX 2080 GPU, for 15 minutes to 1 hour according to different settings.
