\(\bullet\)_Long context_. Long context modeling is an important ability for LLMs, and many studies have explored extending the context windows of LLMs via continually training [235, 238]. With modifications on position embeddings (_i.e.,_ position interpolation) of RoPE-based LLMs [57, 99, 240], CodeLLaMA further extends the context window of LLaMA 2 (2.5T tokens with 4K context window \(\rightarrow\) 20B tokens with 16K context window). LongLLaMA [238] also achieves longer context window with the help of external memory and a unique training objective (1T tokens with 2K context window \(\rightarrow\) 10B tokens with 8K context window).