Inspired by Kocetkov et al. (2023); Lee et al. (2021), we removed duplicates and near-duplicates from the training data using Mou et al. (2023), with default parameters, after which we were left with roughly \(30\)B tokens of high-quality text.