**Group Query Attention.** We train MiniCPM-2.4B without modification on the attention layer. Whereas we apply Group Query Attention (Ainslie et al., 2023) to MiniCPM-1.2B, inspired by Liu et al. (2024), to futher reduce the parameters number.