Task-adaptive pre-training.Continual pre-training of language models on unlabeled data for a given task has been demonstrated to be beneficial for enhancing end-task performance [1; 8; 13]. In scenarios involving domain shift, domain-adaptive pre-training bears similarities to task-adaptive pre-training to some extent. Aharoni et al [1] documented that continual pre-training a model on a similar domain contributes to improved task performance on the target domain. Notably, the work closest to ours is presented in [8], which shows that continual pre-training of language models on both unlabeled task data and augmented unlabeled task data, sampled from the in-domain corpus based on similarity. While these works use task data, we also propose a task agnostic method, ETA-DACP, as task similarity is prohibitively expensive for LLMs.