When the model is trained with MaskedLM using \(r=1\), the model encounters random tokens unrelated to the current context, leading it to disregard the semantics of these tokens, akin to the role of [mask]. In the experiments in Tab. 3, \(r=1\) yields slightly higher results than \(m=1\). While \(r=1\) demonstrates higher levels of randomness in comparison to \(m=1\), it has the potential to better mitigate overfitting. However, on average, the distinction between \(r=1\) and \(m=1\) is marginal for larger datasets. With the exception of using \(r=1\) for GSM8K-7K, we default to utilizing \(m=1\) for other datasets in Tab. 1. We also explore the impact of adding attention masking at the masked positions within each transformer layer, beyond just the input masking. We find that its effectiveness is on par with using input masking alone.