After the release of ChatGPT (OpenAI, 2022), instruction tuning becomes an indispensable ingredient to build the state-of-the-art dialogue agent which has astonishing zero shot capability to a wide range of tasks. The conversational QA capability plays a crucial role in a dialogue agent, yet there has been limited research dedicated to this vital aspect.