The hyperparameters for training of 1.5B, 6B, and 32B models are shown in Table 3. The hyperparameters for training of smaller models are shown in Table 4. The sequence length is 2048 and the optimizer is AdamW [27] with \(\beta_{1}=0.9\) and \(\beta_{2}=0.95\).