When viewing the task of renewing the internal knowledge of LMs as one of CL formulations, pretraining on the original corpus can be considered as a _previous task_, and continued pretraining on new corpus can be considered as the _current task_, the main objective becoming retaining the _time-invariant_ world knowledge gained through initial pretraining while efficiently learning _new_ and _updated_ world knowledge through continued pretraining. Throughout the paper, we let \(D_{0}\) refer to the corpus used for initial pretraining and let \(D_{1}\) denote the new corpus used for continued pretraining.