As shown Figure 1, our Infini-attention computes both local and global context states and combine them for its output. Similar to multi-head attention (MHA), it maintains \(H\) number of parallel compressive memory per attention layer (\(H\) is the number of attention heads) in addition to the dot-product attention.