Following the empirical experiments in scaling laws [15; 21; 16], we further pre-train 28 relatively smaller models with different numbers of training tokens. The model sizes range from 300M, to 540M, 1B, 1.5B, 3B, and to 6B, while the numbers of pre-training tokens range from 33B to 500B. The learning rate schedule is set to reach the minimum at the corresponding token count, which is critical to the optimal performance [21; 16]. The number of tokens used and hyperparameters for all models are shown in Table 4 (Appendix).