In this work, we propose a novel continual learning (CL) formulation named Continual KnowledgeLearning (CKL), where we attempt to renew the internal world knowledge of LMs through continual pretraining on new corpora. We systematically categorize world knowledge into three main categories and make benchmark datasets to measure each of them during CKL: (1) InvariantLAMA for _time-invariant_ world knowledge in LMs that should not be forgotten or altered, (2) UpdatedLAMA for outdated world knowledge that needs to be _updated_ in the LMs, and (3) NewLAMA for _new_ world knowledge that should be injected into the LMs. We also propose a novel metric named FUAR (Forgotten / (Updated + Acquired) Ratio) that can measure the trade-off between forgetting, updating, and acquiring knowledge. Finally, while one might think of implementing contemporary CL methods for this benchmark, we show that CKL has nontrivial differences to traditional CL formulations and require approaches specific to CKL. We find and compare model architectures and training methodologies (Chen et al., 2020; He et al., 2021; Hu et al., 2021; Wang et al., 2021b) from the literature that have shown potential to mitigate forgetting of knowledge gained during pretraining, establishing them as baselines for the CKL benchmark.