Helen Ngo, João G. M. Araújo, Jeffrey Hui, Nicholas Frosst &ndash; Cohere, Toronto, Canada
The One Billion Word Benchmark is a dataset derived from the WMT 2011 News Crawl, commonly used to measure language modeling ability in natural language processing. We train models solely on Common Crawl web scrapes partitioned by year, and demonstrate that they perform worse on this task over time due to distributional shift. Analysis of this corpus reveals that it contains several examples of harmful text, as well as outdated references to current events. We suggest that the temporal nature of news and its distribution shift over time makes it poorly suited for measuring language modeling ability, and discuss potential impact and considerations for researchers building language models and evaluation datasets.
Common Crawl is a repository of web scrapes of the internet updated annually and is often used as a key data source for language models built on the open web [8, 2, 1]. We train benchmark models on three distinct datasets created by selecting data sampled from different years of Common Crawl: 2013 (the year which lm1b was released), 2016, and 2020. [...] Models which are trained on datasets temporally further removed from the lm1b corpus source (i.e. WMT 2011 News Crawl dataset) exhibit higher perplexity than those trained on datasets which are temporally closer.