OverviewInspired by the practice of reference model in document-level filtering, we propose a simple pipeline of token-level data selection, termed "Selective Language Modeling (SLM)". Our method comprises three steps, as depicted in Figure 4. We begin by training a reference model on a curated, high-quality dataset. This model then assesses the loss of each token within the pretraining corpus. In the final phase, we train the language model selectively, focusing on tokens with high excess loss between the training and reference model.