ScalabilityDue to budget constraints, we have only verified the effectiveness of our method on smaller models (<=7B parameters) and smaller datasets (<100B tokens). Smaller models benefit significantly from removing the loss of irrelevant tokens and focusing on important ones. However, it's possible that very large models trained on extensive corpora may naturally develop this inductive bias to compress useful data (_i.e.,_ compressing everything), although it may sounds inefficient for now. Therefore, future works should study whether this selective language modeling technique can scale to very large models and data Kaplan et al. (2020).