Table 10 shows the CKL benchmark performance of GPT-2 models. We report the results averaged over 5 runs with different random seeds. As in Table 2, parameter-expansion methods show robust performance on both IL and NQE, resulting in low FUAR. This shows that these methods are not only effective on the encoder-decoder model but also the decoder-only model as well. One interesting result in Table 10 is that GPT2-MixReview performs the best on IL, with performance even higher than the initial model, which results in the best FUAR of 0 which means no forgetting occurred at all. We suppose that the training strategy of GPT2-MixReview, allowing access to samples of \(D_{0}\) during continued pretraining, would have allowed fast adaptation to knowledge from \(D_{0}\) during the _light-tuning_ phase. Performance of GPT2-MixReview suggests that it makes it possible to regain the original knowledge for decoder-only models even with small tuning steps.