1. We propose a two-stage instruction tuning method and a dataset curation recipe that can largely enhance LLM's capability of integrating user provided or retrieved context for zero-shot conversational QA tasks. We demonstrate that our method significantly outperforms regular instruction tuning or RLHF-based recipes (e.g., Llama-2-Chat). 2. For RAG in conversational QA, we show that fine-tuning state-of-the-art single-turn query retrievers on human-annotated multi-turn QA dataset works as well as utilizing the state-of-the-art LLM-based query rewriting model, i.e., GPT-3.5-turbo (OpenAI, 2022). 3. We build a family of ChatQA models based on Llama2-7B, Llama2-13B, Llama2-70B (Touvron et al., 2023) and a in-house 8B pretrained GPT. We conduct comprehensive study on 10 conversational QA datasets, including 5 datasets with long documents that need retrieval and 3 datasets with tables. In terms of average score, our ChatQA-70B model (54.14) can outperform GPT-3.5-turbo (50.37) and GPT-4 (53.90) without utilizing any synthetic data from ChatGPT models. 4. We study the "unanswerable" scenario, where the desired answer is not included in the provided or retrieved context, thus the LLM can easily hallucinate. We demonstrate that adding a small amount of "unanswerable" samples in instruction tuning can steer the model to generate "can not answer" output when it is necessary, thus largely reduce hallucination. Our ChatQA-70B outperforms GPT-3.5-turbo in this regard, while still has a slight gap compared to GPT-4 (around 3.5%).