Despite its importance, the challenge of renewing the internal world knowledge stored in the parameters of LMs is nontrivial and has only been explored in rather specific settings. For example, recent works have proposed to modify specific target knowledge such as individual facts (De Cao et al., 2021; Zhu et al., 2020; Dai et al., 2021). Dhingra et al. (2021) have addressed LMs as temporal knowledge bases by jointly modeling text with its timestamp. But the problem of renewing the world knowledge of LMs in a more general and scalable way, such as through continual pretraining on a corpus with new knowledge, has not been formally formulated or explored by previous works. Moreover, the community lacks a benchmark that can be used to systematically study how the internal knowledge of LMs changes through the training on new information. Lastly, methodologies to effectively renew the knowledge of LMs at scale have yet to be thoroughly explored.