In order to make a specialist or domain-specific LLM, they need to be trained on domain data. Approaches for building domain-specific LLMs can be categorized into two categories: training domain-specific LLMs from scratch or using continual pre-training existing LLMs with domain data. Most researchers have taken the first approach of building domain-specific LLMs from scratch. Prominent examples are the Med-PaLM family [23; 24] for the medical domain, Galactica for scientific papers [26], and BloombergGPT [31] for finance. Little attention has been paid to building domain-specific LLMs using domain-adaptive continual pre-training, despite being a much cheaper alternative. Notably, PMC-LLaMA [30], a medical LLM was trained through continual pre-training of LLaMA [27] on medical papers. Continual pre-training can also be used for updating a LLM with the latest knowledge in an evolving environment.