1. We introduce a practical and yet powerful attention mechanism - Infini-attention with long-term compressive memory and local causal attention for efficiently modeling both long and short-range contextual dependencies. 2. Infini-attention introduces minimal change to the standard scaled dot-product attention and supports plug-and-play continual pre-training and long-context adaptation by design. 3. Our approach enables Transformer LLMs to scale to infinitely long context with a bounded memory and compute resource by processing extremely long inputs in a streaming fashion.