Table 11 shows the CKL benchmark result of GPT-2 models continually pretrained on CC-RecentNews-Small for 8 epochs with a learning rate of 1e-3. By comparing the results in this table with those in Table 10, which is for models continually pretrained with a learning rate of 1e-4, the results in Table 11 shows worse performance on both IL and NQE. Unlike in Appendix E, increasing the learning rate does not result in better learning of new knowledge. Instead, NQE performance is even worse than GPT2-Initial for GPT2-Vanilla, GPT2-Recadam, and GPT2-MixReview. FUAR is _no gain_ for these cases by the definition of the metric because the denominator has the value of zero. This shows that a large learning rate for continual pretraining may lead to failure: neither retaining old knowledge nor acquiring new knowledge effectively. For parameter-expansion methods, because many parameters including the decoder are frozen during the continual training process, they seem to be less prone to the effect of a large learning rate.