In real applications, there is an increasing demand for long context modeling capacities of LLMs, such as PDF processing and story writing [286]. Many closed-source LLMs provide professional support for long text processing. For instance, OpenAI releases GPT-4 Turbo with a 128K context window, and Anthropic releases Claude 2.1 with a 200K context window. To enhance the long context modeling abilities, there are generally two feasible directions, namely scaling position embeddings and adapting context window. Next, we introduce the two parts in detail.