Figure 1: Infini-attention has an additional compressive memory with linear attention for processing infinitely long contexts. \(\{KV\}_{s-1}\) and \(\{KV\}_{s}\) are attention key and values for current and previous input segments, respectively and \(Q_{s}\) the attention queries. PE denotes position embeddings.