Conversational AI systems for goal-oriented dialogue and task completion have a rich history [49, 50, 51]. The emergence of transformers [52] and large language models [15] have led to renewed interest in this direction. The development of strategies for alignment [53], self-improvement [54, 55, 56, 57] and scalable oversight mechanisms [58] have enabled large scale deployment of such conversational systems in the real world [59, 16]. However, the rigorous evaluation and exploration of conversational and task-completion capabilities of such AI systems remains limited for clinical applications, where studies have largely focused on single-turn interaction use cases such as question-answering or summarization.