\(\bullet\)_\(\Lambda\)-shaped context window._ Some prior work has revealed that LLMs tend to allocate greater attention weights to the starting and nearest tokens among all previous tokens [297, 298], so called the _"lost in the middle"_ phenomenon [299]. Based on this observation, LM-Infinite [300] and StreamingLLM [298] propose to employ a "\(\Lambda\)-shaped" attention mask, which selectively preserves the initial tokens and the nearest tokens that each query can attend to and then discards any tokens beyond this scope. Experiments demonstrate that this method can facilitate extra-long text generation with a fixed memory [298]. However, it may struggle to model the long-range dependency in prompts, since it cannot effectively utilize the information from the discarded tokens [298].