* **Findings on task dynamic prediction:** Within the training process of LLMs, we observe that the dynamics of existing downstream tasks within a domain can predict the dynamics of unseen tasks. This suggests that a model's performance on known tasks can inform us about how it might perform on similar, yet unseen tasks within the same domain. (Section 4.1.1)
* **Findings on cross-domain promotion:** Similar to the human cognitive process, the enhancement of different abilities across various domains progresses from basic to advanced levels following curriculum learning. The curriculum between cross-domain tasks can guide the training direction of models and the insights gained from one domain can potentially promote the learning process of other domains. (Section 4.1.2)
* **Findings on the effect of training strategies, model architecture, etc. :** Based on the results of several 7b-scale models (Baichuan2-7b, DeepSeek-7b, OpenLLaMA-7b, and Amber-7b), we comprehensively analyze the effect of training strategies, dataset quality, and model architecture in training LLMs. For example, we observe that the training dataset, learning rate adjustments, batch size, and regularization techniques play a significant role in the learning efficiency in the early training stage (Section 4.2.1). * **Findings on the effect of model scale on reasoning tasks:** Based on the results of Deepseek-7b, 34b and 67b, we observe that the model size and complexity significantly influence its ability on these reasoning tasks. Nonetheless, employing particular strategies can enhance smaller-scale models to obtain similar performance on commonsense reasoning when compared with their larger counterparts. (Section 4.2.2)
* **Findings on the scaling law:** (1). We observe that larger training datasets lead to improved model performance on various benchmarks, which demonstrates the effect of extensive training data in training LLMs. However, the benefit of additional data diminishes as datasets grow, suggesting an approaching limit to performance gains. (2). We observe that the accuracies of the scaling law (Hoffmann et al., 2022) vary a lot across different models, which indicates that factors such as model architecture and computational complexity significantly influence scaling efficiency. Notably, some models demonstrate better alignment with the law, suggesting potential advantages in data utilization and learning efficiency. (3). Although the scaling law can provide a useful perspective on the impact of training data size, the actual performance scaling is nuanced, which reflects the complex interplay between data volume, model architecture, and computational strategies. This also highlights the importance of continued research into scaling laws and model optimization to maximize learning outcomes from available data (Section 5).