Here, \(\mathcal{L}_{\text{CLM}}(\theta)\) represents the loss function parameterized by model \(\theta\). \(N\) is the length of the sequence, \(x_{i}\) is the \(i\)-th token in the sequence, and \(x_{<i}\) represents all tokens before the \(i\)-th token. In contrast, Selective Language Modeling (SLM) trains the language model with a focus on tokens that exhibit a high excess loss when compared to the reference model.