* **Knowledge acquisition** QA accuracies on test questions measured by exact match. * **Perplexity of documents** We compute perplexity (PPL) on all tokens within the documents. * **Knowledge retention** We approximate the retention of accumulated knowledge during pre-training by assessing the QA accuracy on the Natural Questions (NQ) dataset. NQ was released in 2019, and primarily includes questions based on Wikipedia articles from that time.