To mask the input sequence, we follow <cite>(Baevski et al., 2019b)</cite> and randomly sample p = 0.05 of all tokens to be a starting index, without replacement, and mask M = 10 consecutive tokens from every sampled index; spans may overlap.