The primary limitation of TACP lies in its focus on constructing task-specific LLMs instead of foundation LLMs, owing to the sole use of unlabeled task data for training. While DACP uses a much larger domain corpus, it is prohibitively expensive. To address these limitations, we propose two approaches: _Efficient Task-Similar Domain-Adaptive Continual Pre-training_ (ETS-DACP) and _Efficient Task-Agnostic Domain-Adaptive Continual Pre-training_ (ETA-DACP). While ETS-DACP aims to build foundation LLMs for a set of tasks by tailoring the DACP to emphasize the significance of these tasks, ETA-DACP is more general and selects the most informative samples from the domain corpus for continual pre-training.