Many works study the training dynamics of LMs from different perspectives. Carlini et al. (2022) quantifies memorization across model sizes and the frequency of data duplication. Tirumala et al. (2022) finds that larger LMs memorize training data faster with less overfitting. Xia et al. (2023) show that perplexity is more predictive of model behaviors than other factors. Dery et al. (2022) studies end-task aware pre-training using classification tasks and RoBERTa models. Our work differs in that we specifically focus on the capacity of recalling and generalizing information from a seen document to answer questions.