**Language Modeling.** The language modeling task (LM) is the most commonly used objective to pre-train decoder-only LLMs, _e.g._, GPT3 [55] and PaLM [56]. Given a sequence of tokens \(\mathbf{x}=\{x_{1},\dots,x_{n}\}\), the LM task aims to autoregressively predict the target tokens \(x_{i}\) based on the preceding tokens \(x_{<i}\) in a sequence.