However, despite thorough document-level filtering, high-quality datasets still contain many noisy tokens that can negatively affect training, as illustrated in Figure 2 (Upper). Removing such tokens might alter the text's meaning, while overly strict filtering could exclude useful data (Welbl et al., 2021; Muennighoff et al., 2024) and lead to biases (Dodge et al., 2021; Longpre et al., 2023). Furthermore, research indicates that the distribution of web data does not inherently align with the ideal distribution for downstream applications (Tay et al., 2022; Wettig et al., 2023). For example, common corpus at the token level may include undesirable content like hallucinations or highly ambiguous tokens that are hard to predict. Applying the same loss to all tokens can result in wasted computation on non-beneficial tokens, possibly limiting LLM's potential to merely mediocre intelligence.