\(\bullet\)_External memory._ It has been shown that a relatively small subset of tokens can effectively capture the majority of attention patterns in a Transformer [301], _i.e.,_ the top-\(k\) attention keys can well approximate the original full attention. Therefore, a number of studies propose to store the past keys in external memory and utilize a \(k\)-NN search method to retrieve the \(k\) most relevant tokens for generation [302, 301, 238]. For a decoder model, it typically employs one certain layer to access these top-\(k\) external tokens, while still adopts the normal context window in the rest layers [302, 238].