In this section, we give a brief overview of the BERT<cite> (Devlin et al., 2019)</cite> pretraining approach and some of the training choices that we will examine experimentally in the following section.