Factual knowledge stored in the parameters of LLMs can be accessed and applied to answering questions through prompting without additional training Brown et al. (2020); Petroni et al. (2019); Jiang et al. (2020); Roberts et al. (2020). With additional instruction-tuning (also known as supervised fine-tuning) on high-quality data Sanh et al. (2022); Wei et al. (2022), knowledge seems to be more effectively elicited from LLMs. However, when LLMs correctly answer a question, the source of the knowledge is unclear due to the diversity of the pre-training data. For instance, when answering the question "where is the world's largest ice sheet located", do LLMs derive their response by recalling and generalizing information from a seen document about the Antarctic ice sheet, or do they merely repeat answers from similar questions encountered in the training data? This distinction is crucial, as the former scenario implies an ability to comprehend documents and effectively store knowledge within parameters in a way that can be elicited later, whereas the latter is mere rote memorization.