In all experiments, we use the AMSGrad [1]} variant of AdamW [2]} optimizer with \(\beta _1=0.9\) , \(\beta _2=0.98\) , L2 weight decay of 1e-4. We use a learning rate warm-up for the first 6% of iterations followed by a linear decay of learning rate.
