@inproceedings{
que2024dcpt,
title={D-{CPT} Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models},
author={Haoran Que and Jiaheng Liu and Ge Zhang and Chenchen Zhang and Xingwei Qu and Yinghao Ma and Feiyu Duan and ZhiqiBai and JiakaiWang and Yuanxing Zhang and Xu Tan and Jie Fu and Jiamang Wang and Lin Qu and Wenbo Su and Bo Zheng},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=JzKFN5fWOk}
}
Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model’s fundamental understanding of specific downstream domains (e.g., math and code). For the CPT on domain-specific LLMs, one important question is how to choose the optimal mixture ratio between the general-corpus (e.g., Dolma, Slim-pajama) and the downstream domain-corpus. Existing methods usually adopt laborious human efforts by grid-searching on a set of mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio is optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law of the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs for LLMs of different sizes. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs on limited experiments. Moreover, we also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1\% of the normal training costs) are needed for the target domains. Comprehensive experimental results on six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law.
D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
Haoran Que Jiaheng Liu Ge Zhang Chenchen Zhang Xingwei Qu Yinghao Ma Feiyu Duan ZhiqiBai JiakaiWang Yuanxing Zhang Xu Tan Jie Fu Jiamang Wang Lin Qu Wenbo Su Bo Zheng