In this paper, we introduce Adapt-LLM, a LLM which learns to discern when additional context is necessary for answering a question, rather than relying solely on its parametric memory. Adapt-LLM is the result of fine-tuning a base LLM on an open-domain question answering dataset that has been modified to differentiate between questions answerable with the LLM's parametric memory alone and those requiring supplementary context. To construct these training datasets, we initially subject the base LLM to zero-shot evaluation to determine its accuracy in answering questions. For questions where the model's response is incorrect, we train the LLM to generate a special token, \(\langle\text{RET}\rangle\), indicating the need for additional context.