on understanding physical laws, HellaSwag evaluates how well they predict story outcomes, BoolQ evaluates their comprehension and inference skills through yes/no questions based on text passages, WinoGrande examines their handling of language nuances and context-based ambiguity resolution, and CommonsenseQA assesses their general knowledge and reasoning. These datasets measure the most fundamental capabilities of the models. As shown in Figure 2, within each model, the accuracy on these datasets rapidly increases in the early stages of training (before 300B tokens) and gradually reaches a plateau. This trend is particularly evident in the checkpoints before 100B tokens for the Amber-7b model.