In this paper, we build a family of ChatQA models, varying in model sizes from 7B to 70B. Comprehensive evaluations on 10 conversational QA datasets show that our best ChatQA-70B model can remarkably outperforms GPT-3.5-turbo and perform on par with GPT-4 without using any synthetic data from ChatGPT models. In addition, we demonstrate that fine-tuning a single-turn query retriever using our curated conversational QA data performs comparable to the state-of-the-art LLM-based query rewriting model, without the need of extra computational time and potential API cost from rewriting. Furthermore, we show that incorporating a small amount of "unanswerable" samples can significantly enhance our model's capability to handle scenarios where answers are unavailable. The unanswerable case evaluation highlights that our best model ChatQA-70B only has a slight gap compared to GPT-4.