Dataset ConstructionThe data for continual pretraining, CC-RecentNews, is constructed using news-please (Hamborg et al., 2017). InvariantLAMA is constructed by manually selecting 28 _time-invariant_ relations from T-Rex (Elsahar et al., 2018). For UpdatedLAMA and NewLAMA, we use Amazon Mechanical Turk (mturk)4 for crowd-sourcing Human Intelligent Tasks (HITs). The process requires selecting answerable questions from a list of questions generated by the model introduced in Lewis et al. (2021) and converting them into clozse sentences. We have also separately hired 11 experts to verify the correctness and search the C4 database to categorize each instance following our definition of _updated_ and _new_. NewLAMA-Easy is constructed at a larger scale through a two-phase mturk process where sentences selected from articles containing new information are decontextualized and paraphrased5 before being masked, verified and converted to corresponding questions. The constructed dataset statistics are in Table 1. Important details about the data construction pipeline, examples, and more fine-grained statistics are provided in Appendix B.