In this paper, we propose Continual Knowledge Learning (CKL), where we establish benchmark datasets and metrics, and explore methodologies towards continual knowledge learning of an ever-changing LM. We find that parameter-expansion methods show the most robust performance throughout all of the experimental settings, which nevertheless has severe memory inefficiency and that seeing the same data often is a critical cause of forgetting. We also discuss several other interesting results of which we leave further exploration to future studies. To this end, we suggest the community to explore CKL for the better design of an ever-changing LM.