position, and the input for the second forward pass incorporates some of the predicted tokens from the first pass. In determining which tokens to replace, we employ a sampling strategy characterized by the parameter \(\tau\), representing temperature; a higher \(\tau\) introduces more randomness. At lower temperatures, there is a higher likelihood of replacing tokens with their synonyms, as opposed to random replacement. MaskedLM. In our main experiments, we implement MFT by this method, adhering to the methodologies of prior Masked Language Models as (Bao et al., 2020; Devlin et al., 2019). Unlike encoder-based models (Devlin et al., 2019), which typically predict the current token at masked positions, we predict the next token at masked positions (Xie et al., 2016; Chen et al., 2023). Upon determining the positions for masking, we are presented with two options: employing a [mask] token or substituting with a random token. The probability of selecting the [mask] token is denoted by \(m\), and \(r\) represents the probability of opting for a random token. Su et al. (2022) set \(r\) to 0, while Devlin et al. (2019) set \(r\) to 0.1.