We stipulate that we can form an optimal set \(\mathcal{D}^{*}\) by selecting a portion of the domain data that is much closer to the task data (red) given by the blue region based on intuition before. We refer to this as _Efficient Task-Similar Domain-adaptive Continual Pre-training_ (ETS-DACP). Fine-tuning LLMs can take a good amount of instructions, which are quite costly to create. ETS-DACP directly addresses this situation by using the relatively limited unlabeled task data to sample similar samples from the larger pool of pre-training domain corpus. We are motivated by prior research showing that unsupervised training on tokens that closely align with the target domain and tasks can lead to improved performance [8, 1, 6]. Therefore, we hypothesize that continual pre-training LLMs on the unlabeled task data can be beneficial for target task performance as it adapts the model to the distribution of task tokens.