We use similarity between embeddings of task data and domain corpus samples to perform data selection. This allows us to select a subset from the domain corpus that closely resembles the distribution of task data. To quantify document-level task similarity, we employ cosine similarity between the document embedding and task data embedding. Prior works like [13] calculate embeddings from language model (RoBERTa) for a given unlabeled sample twice, which is not practical for LLMs. It takes a forward pass to compute the embeddings using LLM over entire corpus, or 25% of compute of using to train the pre-train the LLM over the entire corpus. We compute embeddings using the _Spacy_ model [9]. This approach allows us to cost-effectively measure the alignment between task-specific information and the financial corpus, enabling more focused and targeted pre-training.