**Optimizer.** The Adam optimizer [328] and AdamW optimizer [329] are widely utilized for training LLMs (_e.g.,_ GPT-3), which are based on adaptive estimates of lower-order moments for first-order gradient-based optimization. Commonly, its hyper-parameters are set as follows: \(\beta_{1}=0.9\), \(\beta_{2}=0.95\) and \(\epsilon=10^{-8}\). Meanwhile, the Adafactor optimizer [330] has also been utilized in training LLMs (_e.g.,_ PaLM and T5), which is a variant of the Adam optimizer specially designed for conserving GPU memory during training. The hyper-parameters of the Adafactor optimizer are set as: \(\beta_{1}=0.9\) and \(\beta_{2}=1.0-k^{-0.8}\), where \(k\) denotes the number of training steps.