The main contributions of this paper are threefold. Firstly, we curate a large-scale financial corpus comprising 16 billion words sourced from financial datasets. Secondly, our experiments demonstrate the promise of building domain-specific LLMs through continual pre-training, further validating and extending the findings obtained from smaller language models [8]. This finding provides insights for building domain-specific LLMs with lower costs, as an alternative to expensive pre-training from scratch. Our results indicate that continual pre-training maintains the same open-domain performance as the original foundation model. Lastly, we propose two Efficient Domain-adaptive Continual Pre-training methods as a more efficient approach to vanilla continual pre-training. Our novel approach deploys data selection strategies that can achieve better performance with a fraction of the cost of the domain-adaptive continual pre-training.