predicted [MASK] is not considered. Additionally, the introduction of [Mask] as a special marker in the input to replace the original Token creates inconsistency between the data in the Pre-Training stage and the Fine-Tuning stage, where [Mask] is not present. Self-encoding language models are commonly used in RALM architectures for natural language understanding (NLU) tasks.