In this paper, we demonstrate the efficacy of domain-adaptive continual pre-training for developing domain-specific LLMs. Our results in the finance domain show that domain-adaptive continual pre-training improves the LLMs' performance on financial tasks. Domain-adaptive continual pre-training enables the LLMs to acquire new knowledge in the financial domain at a much lower cost.