**Long-context continual pre-training.** There is a line of work that extends the do-product attention layers and continues to train LLMs for long-context (Xiong et al., 2023; Fu et al., 2024). The attention extensions include incorporating sparsity into the attention layer (Chen et al., 2023; Ratner et al., 2022; Mohtashami and Jaggi, 2024) as well as manipulating the position encodings (Chen et al., 2023; Peng et al., 2023) Although the position encoding-based methods such as position interpolation techniques (Chen et al., 2023) can be data efficient as they only adjust the positional bias in the attention layer, they are still costly for inference.