Recent works have shown that Transformer-based (Vaswani et al., 2017) Large Language Models (LLMs) store and retrieve factual information in their parameters to complete simple prompts such as _"The mother of Stevie Wonder is"_(Petroni et al., 2019; Meng et al., 2022; Geva et al., 2021, 2022, 2023; Zhu and Li, 2023). In addition, LLMs have demonstrated remarkable _in-context_ reasoning abilities when the necessary information is explicitly given as part of the input (Wei et al., 2022). For example, models can infer "Lula" as a possible completion of _"The mother of Stevie Wonder is Lula. The singer of 'Superstition' is Stevie Wonder. The mother of the singer of 'Superstition' is"_. These findings raise a question: Do LLMs retrieve factual information stored in their parameters and perform _latent multi-hop reasoning_ when the information to reason from is _not_ given as a part of the input? For instance, when LLMs process the two-hop prompt _"The mother of the singer of 'Superstition' is"_, do they (1) figure out that "the singer of 'Superstition'" refers to Stevie Wonder and (2) use their knowledge of who Stevie Wonder's mother is to complete the prompt?