**Denoising Autoencoding.** In addition to conventional LM, the denoising autoencoding task (DAE) has also been widely used to pre-train language models [82, 24]. The inputs \(\mathbf{x}_{\setminus\hat{\mathbf{x}}}\) for DAE task are corrupted text with randomly replaced spans. Then, the language models are trained to recover the replaced tokens \(\tilde{\mathbf{x}}\).