Although numerous research about instruction tuning has been conducted, a few work focused on improving RAG or context awareness generation for QA. Lin et al. (2023) introduced a retrieval-augmented instruction tuning method, which appends top-\(k\) retrieved chunks for LLM fine-tuning. Wang et al. (2023) applied instruction tuning after retrieval-augmented pretraining. In contrast, we propose a two-stage instruction tuning method to improve generation with retrival or provided context. We find that appending top-\(k\) retrieved chunks for LLM fine-tuning does not help for a wide range of conversation QA tasks (see SS6.3 for details).