biographies led to strong generalization when answering questions about the remaining half of biographies, which resembles setting 1 in Fig. 4. In contrast, training on biographies and QA pairs sequentially failed. However, the key contributor to the success remains uncertain because the data were blended together, and it is unclear how to apply this practically to absorb knowledge from new documents. Inspired by our observation of the different difficulty levels between QA pairs and documents, and the finding from Zhu and Li (2023), we hypothesize that _it is beneficial to deliberately expose LLMs to instruction-tuning data before continued pre-training so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed._ We refer to this as **pre-instruction-tuning (PIT)** and study various implementations of PIT prior to continued learning (SS 5.1), followed by detailed ablations identifying the keys contributor to performance (SS 5.2 and SS 5.3), and finally assess how well PIT performs across domains (SS 5.4). We adhere to the hyperparameters outlined in SS 3.2 and perform PIT for 3 epochs unless specified otherwise.