Baselines MethodsWe pre-train LLMs of (i) Half of C4, and the (ii) Full C4 corresponding to approximately 85 Billion and 170 Billion real tokens respectively (Raffel et al., 2020). We also pre-train our own models on (iii) 160 Billion and (iv) 320 Billion tokens of the RefinedWeb Dataset (Penedo et al., 2023). Additionally, we also compare with the (iv) Pythia-1.4B model that was trained on the Pile (Gao et al., 2020). This dataset is no longer publicly available, hence we utilize a pre-trained model. Finally, we also compare with the recent (v) TinyLlama model (Zhang et al., 2024) that was trained for 3 epochs on 1 Trillion tokens of data from SlimPajama (Shen et al., 2023) and StarCoder (Li et al., 2023a).