respectively. We compare how CKL methods for T5 perform on IL, NLE\({}_{\text{P1}}\), and NLE\({}_{\text{P2}}\) when continually pretrained entirely on Small for 5k steps (8 epochs), and when sequentially pretrained on Small-P1 and then on Small-P2 for 2.5k steps (8 epochs) each. In the scenario Small-P1\(\rightarrow\)Small-P2, there are two CKL phases where \(D_{0}\) is C4 and Wikipedia, \(D_{1}\) is Small-P1, and \(D_{2}\) is Small-P2. The rest of the configurations are set identical with the main experiments.