We use three datasets for training and validation: SlimPajama (Soboleva et al., 2023), German CommonCrawl (Laippala et al., 2022), and Pile (Gao et al., 2020). For all datasets, use the same tokenizer as Black et al. (2022) trained specifically on the Pile. To create our training set for SlimPajama, we randomly sub-sample the dataset (606B Total Tokens) to form a \(\sim\)299B token subset (see Table 1) that is of comparable size to Pile. We also further sub-sample this SlimPajama subset to create three \(\sim 100\)B token splits of the dataset (see Sec. 7.4 for details). To create the SlimPajama validation set we simply tokenize the default validation set that has been extensively deduplicated (Soboleva et al., 2023). To create the German training and validaiton sets, we split and tokenized the German Common Crawl scrape, available as part of the Oscar Dataset (Laippala et al., 2022), into a 195.43B token training set and a 982.6M token validation set. The pile dataset comes pre-shuffled and mixed, we simply used the default training and validation sets. The training set is \(\sim 330\)B tokens total, though in our experiments we only train on a 300B token subset.