In this work, we explore the following: 1) Is domain-adaptive continual pre-training helpful in building domain-specific LLMs?; 2) Can we employ data selection strategies for a more effective domain-adaptive continual pre-training?; and 3) Does domain-adaptive continual pre-training hurt LLM's open-domain capabilities? We answer these questions in the confines of finance domain by training a continually pre-trained model, FinPythia, built on top of Pythia [4].