In this section, we first show the main experimental results for the CKL Benchmark. Then, since multiple steps of continual knowledge learning, i.e., CKL are needed for training a true, ever-changing LM, we explore the effects of multiple CKL phases as well as how epochs, corpus size, and the total number of training steps affect CKL. We further explore how learning rates affect CKL in Appendix E, how continual pretraining on \(D_{1}\) affects the performance of KILT tasks which re quire knowledge from \(D_{0}\) in Appendix F, how CKL methods transfer across LM architectures in Appendix G, and how the prediction outputs change during CKL in Appendix H.