Pre-instruction-tuning w/ QA onlyWe start with exposing instruction-tuning data before continued pre-training on documents--training on topically related QA pairs before training on test documents (Fig. 4 5). This can be directly compared with the continued pre-training setting (Fig. 4 5). The intuition is that questions help LLMs recognize key types of information, enabling LLMs to focus on important information during pre-training on subsequent documents, even though the questions are not directly tied to the documents. For example, training on a question like "who handled the editing of Oppenheimer" could help LLMs pay attention to screenwriters when training on new documents like "Barbie". As shown in Tab. 1, this method outperforms continued pre-training, especially on larger LLMs (27.6%/41.7% \(\rightarrow\) 28.6%/49.7% for 7B/70B). The ablation that trains on QA data after training on documents ("instruction-tuning w/o train doc" in Tab. 2) is ineffective, confirming the importance of training on questions as a warm-up before encoding documents.