Activation FunctionWe use SwiGLU [68] as Yi's post-attention layer, reducing its activation size from \(4h\) to \(8/3h\) (\(h\) denotes hidden size) to be consistent with the normal post-attention layer. This adjustment also compensates for the reduction in parameter resulted from GQA, making the overall parameter count compatible of existing 7B and 34B models.