Common Crawl is a repository of web scrapes of the internet updated annually and is often used as a key data source for language models built on the open web [1]}, [2]}, [3]}. We train benchmark models on three distinct datasets created by selecting data sampled from different years of Common Crawl: 2013 (the year which lm1b was released), 2016, and 2020. Each benchmark model followed a standard decoder-based Transformer architecture [1]}, [5]} with 128M parameters, and was trained on a 10 GB randomly-sampled subset of Common Crawl for 70k steps on Cloud TPU v2 chips using the Adam optimizer with initial learning rate = 0.0001, batch size = 1024, sequence length = 1024, embedding dimension = 1024, layers = 6, attention heads = 4. Figure REF  demonstrates the negative impact of distribution shift over time. Models which are trained on datasets temporally further removed from the lm1b corpus source (i.e. WMT 2011 News Crawl dataset) exhibit higher perplexity than those trained on datasets which are temporally closer.
<FIGURE>