Scaling up model parameters and dataset size has consistently elevated the next-token prediction accuracy in large language models, yielding significant advancements in artificial intelligence (Kaplan et al., 2020; Brown et al., 2020; OpenAI, 2023; Team et al., 2023). However, training on all available data is not always optimal or feasible. As a result, the practice of data filtering has become crucial, using various heuristics and classifiers (Brown et al., 2020; Wenzek et al., 2019) to select training documents. These techniques significantly improve data quality and boost model performance.