**Effects of Epochs, Corpus Size, and Total Number of Training Steps in CKL on Forgetting** Figure 3 shows the result of T5-Vanilla and T5-Kadapters during continued pretraining in different scenarios from Table 2 and 3, where each point in the graph represents the performance of IL after every epoch. Comparing Main (4 epochs) and Small (8 epochs) in Figure 3 (a) T5-Vanilla, we can see that more forgetting occurs in Small, even though trained for five times less number of global training steps. This phenomenon is further highlighted when comparing results from Small-P1 (8 epochs) which shows the most amount of forgetting despite being trained for ten times less number of global training steps. While the overall drop is much mitigated in Figure 3 (b) T5-Kadapters, we observe the same trend between each scenario which goes to show how critical observing the same data repeatedly during continued pretraining is for causing forgetting.