Footnote 14: GPT-2 was initially pretrained on WebText (Dec 2019), which consists of 8 million documents with Wikipedia pages excluded. In order to measure the performance on InvariantLAMA constructed from Wikipedia, we continually pretrain GPT-2 on a subset of Wikipedia (May 2020) for 14k global training steps before CKL.