As is the case with<cite> Devlin et al. (2018)</cite> , we find that choosing a batch size of 16, learning rate of 2×10 −5 , and MSL of 512 tokens yields optimal performance on the validation sets of all datasets.