When training on documents, we prepend a <bos> token and compute the standard next-token prediction loss by averaging over all tokens in the document: \(L_{\mathbf{d}}=-\sum_{t}\log P(\mathbf{d}_{t}|\mathbf{d}_{<t})/|\mathbf{d}|\).5 When training on QA pairs, we compute the average negative log-likelihood loss only on tokens in the answer given the question as the prefix: \(L_{\mathbf{a}}=-\sum_{t}\log P(\mathbf{a}_{t}|\mathbf{q},\mathbf{a}_{<t})/|\mathbf{a}|\). Fig. 3 presents an example document alongside QA pairs, where tokens used for computing losses are highlighted.