We report a boost on financial benchmarks [33] after continual pre-training on domain data of size 8% of what Pythia was trained on as an answer to the first question. We also observe an evidence of latest financial domain knowledge acquisition in FinPythia during qualitative analysis. To answer the second question, we propose two simple data selection techniques, task-aware _Efficient Task-Similar Domain-Adaptive Continual Pre-training_ (ETS-DACP) and _Efficient Task-Agnostic Domain-Adaptive Continual Pre-training_ (ETA-DACP). These methods outperform the performance of domain-adaptive continual pre-training with just 10% of selected domain data or 0.8% of Pythia's training corpus. We use three metrics for data selection: similarity, perplexity, and token type entropy. While similarity needs task data as seed, the latter two metrics are task-agnostic. To answer the third question, we benchmark these continually pre-trained models on four open-domain standard tasks like MMLU and TruthfulQA. We observe no significant performance change, indicating that LLM retains its general capabilities while adapting to the domain.