Pretraining ConfigurationWe utilize the T5 initially pretrained on C4 (April 2019) and continually pretrained with salient span masking (Guu et al., 2020) on Wikipedia (May 2020) as initialization. We use the checkpoints from Wolf et al. (2020). We also perform the SSM objective during CKL because it was shown to help LMs "focus on problems that require world knowledge" (Guu et al., 2020; Roberts et al., 2020).