We train a standard BERT model (Devlin et al., 2018; with only the masked language modeling task on each set of inputs in the same way as described in <cite>(Baevski et al., 2019b)</cite> , namely by choosing tokens for masking with probability of 0.05, expanding each chosen token to a span of 10 masked tokens (spans may overlap) and then computing a cross-entropy loss which attempts to maximize the likelihood of predicting the true token for each one that was masked ( Figure  1a ).