Our domain-adaptive continual pre-training can be viewed from the lens of unsupervised domain adaptation [7]. Our source data is the large unsupervised domain corpus, while the target data is the target task data. With pre-training, we do not have control over the alignment with task training data itself; our idea is that by aligning with the domain during pre-training, we could align the LLM with the task. This intuition is backed by evidence of LLM pre-training helping the performance over open domain tasks. We use the generalization bound from [7; 2] since our problem is similar to unsupervised domain adaptation. Consider a hypothesis space \(\mathcal{H}_{p}\) with \(f\in\mathcal{H}_{p}\); generalization errors on source \(\mathcal{D}\) and task data \(\mathcal{T}\) as \(\epsilon_{\mathcal{D}}\) and \(\epsilon_{\mathcal{T}}\), respectively.