Given these distinctions, for LLaMA models, we use the notation LLaMA-\(\ell\)-\(h\) for \(\ell\) layers, \(h\) heads, and \(64h\) hidden dimensions; we omit group-query attention as LLaMA recommends it only for its 70B model. For Mistral, denoted as Mistral-\(\ell\)-\(h\), we enable group-query attention with 4 groups if \(h=0\pmod{4}\), 1 group for odd \(h\), or 2 groups otherwise.