**Validation Set Perplexity**. Our validation sets mainly come from [59], which includes validation sets derived from subsets of the Pile [16] such as CommonCrawl, DM Mathematics, HackerNews, OpenSubtitles, OpenWebText2, Project Gutenberg, USPTO, Wikipedia. We also include a validation set obtained from the PushShift.io Reddit dataset [4] (which we refer to as _redditflattened_). In addition, we measure perplexity on a validation set obtained from a train-validation split of our source dataset _CC-dedup_, and a validation set from C4 [41].