**Activation Functions.** To obtain good performance, activation functions also need to be properly set in feed-forward networks. In existing LLMs, GeLU activations [270] are widely used. Specially, in the latest LLMs (_e.g._, PaLM and LaMDA), variants of GLU activation [262, 271] have also been utilized, especially the SwiGLU and GeGLU variants, which often achieve better performance in practice [266]. However, compared with GeLU, they require extra parameters (about 50%) in the feed-forward networks [272].