We perform extensive experiments with an encoder-decoder model, T5 (Raffel et al., 2019), a large LM (\(\sim\) 737M params) initially pretrained on April 2019 dump of C4 and May 2020 dump of Wikipedia (thus \(D_{0}\) in our experiments) with salient span masking (SSM). The details of the pretraining, continual pretraining, and evaluation configurations are in Appendix C. We establish the following methods as the baselines for the CKL benchmark and categorize them into _regularization_, _rehearsal_, and _parameter-expansion_ methods. The specific hyperparamters used for the implementation of each method are detailed in Appendix D.