FormulationWe first formalize the problem. We are given an unlabeled domain pre-training corpus, \(\mathcal{U}\) represented by green region in Figure 1. Next, we can take two scenarios: absence or presence of an unlabeled task corpus. The first scenario of the presence of a task corpus, which can be a single or group of tasks, \(\mathcal{T}\) is depicted as the red region in Figure 1. Typically, the task corpus is a subset of the domain corpus, \(\mathcal{T}\subset\mathcal{U}\), with \(|\mathcal{U}|>>|\mathcal{T}|\). The goal of data selection is to select a subset, \(\mathcal{D}\subset\mathcal{U}\), that is most helpful for pre-training the LLM model. We also assume that the selected domain corpus subset is much larger than the task corpus, \(|\mathcal{D}|>>|\mathcal{T}|\), as is a typical case.