Second, SIQA, ARC (both Easy and Challenge), OpenBookQA and MMLU target more advanced cognitive abilities compared to datasets like PIQA, HellaSwag, WinoGrande and CommonsenseQA, with accuracy improvements on these benchmarks typically occurring later in the training process. SIQA tests social comprehension, ARC spans basic to complex scientific reasoning, OpenBookQA requires factual integration with textual understanding and MMLU measures knowledge application across multiple disciplines. These datasets emerge as crucial in the mid-training phases, shifting from basic commonsense to intricate reasoning and domain-specific knowledge application. This progression underscores a layered approach in AI training, moving from foundational understanding to higher-order cognitive skills.