Specifically, for questions where the LLM's response is incorrect, we annotate them with a special token, \(\langle\text{RET}\rangle\), indicating the need for additional context. Subsequently, we utilize these annotations to construct a new dataset tailored for training purposes, where we teach an LLM to answer directly if it is confident about the answer or to require context it believes is useful for answering the question (see Figure 1). Our hypothesis is that through this training process, the LLM learns to use an IR system when it needs extra context to answera question, thus we name it Adapt-LLM.