Language models pretrained on text corpora from the open web will not capture knowledge about the world from outside the temporal window of the corpus curation process. For example, GPT-3 was trained on scrapes of Common Crawlhttps://commoncrawl.org/the-data/ from 2016 to 2019 [1]}, and cannot reliably generate information about current events which happened after the 2019 cutoff window.
lm1b was released in 2013 using a web scrape from 2011, which means that models evaluated on lm1b today are attempting to represent world knowledge about events which happened at least a decade ago. These events no longer represent the world as it exists today, and measuring language models on their ability to generate text from news articles from 2011 inherently penalizes models which learn more recent, up-to-date world representations.
