Pre-training procedure Similar to<cite> (Devlin et al., 2019)</cite> we use Adam with learning rate of 1e − 4, β 1 = 0.9, β 2 = 0.999, L2 weight decay of 0.01, learning rate warmup over the first 10, 0000 steps, and linear decay of the learning rate.