Similar to the standard MHA, for the multi-head Infini-attention we compute \(H\) number of context states in parallel, and concatenate and project them for the final attention output \(O\in\mathbb{R}^{N\times d_{mult}}\):