We perform experiments with GPT-2 Large (\(\sim\) 774M params) (Radford et al., 2019) initially pre-trained on WebText and Wikipedia14 (\(D_{0}\)) and continually trained on CC-RecentNews-Small, i.e., Small (\(D_{1}\)) for 8 epochs. For continued pretraining, we use the common teacher-forcing pretraining objective. The initial learning rate for the continued pretraining stage is empirically chosen as 1e-4 (results with learning rate as 1e-3 are shown in Appendix G.1). After continued pretraining, we apply _light-tuning_, a process denoted for finetuning the model for only one epoch on a small portion of data similar to the evaluation set. Training on a single epoch constrains the model to barely adapt to the input-output form of the data and not to learn the knowledge in tuning samples, mitigating the problem suggested by Lewis et al. (2020).