In addition to the public available datasets, one of the key elements for stage-2 is to obtain a high-quality document-grounded conversational QA dataset. We create a human-annotated conversational QA (called HumanAnnotatedConvQA) dataset only consisting of 7k dialogues. To build this dataset, we first collected 7k documents covering diverse topics from Internet. Then, we instruct the annotator to act both as a curious user asking questions (and follow-up questions) about the document and as a agent to give the answers. We create a multi-turn conversation for each document, resulting in a total of 7k conversational QA dialogues with an average of 5 user-agent turns per dialogue. Details of data collection guideline can be found in Appendix G.