In addition to human evaluations, we implemented model-based auto-evaluation methods as economical consistent alternatives to specialist assessments. These techniques were employed to evaluate both dialogue quality and diagnostic accuracy of the OSCE agent. To establish the validity of our auto-evaluation methods for assessing dialogue quality, we initially focused on a subset of four evaluation axes from the PACES rubric(Table A.2) that were assessed by both the patient actors and the specialist physicians. The auto-evaluation, which uses a self-CoT strategy (details described in Section A.9) with AMIE to rate dialogues, was in good alignment with human raters and comparable to the inter-specialist agreement on these criteria. For the auto-evaluation of differential diagnoses, we leveraged another LLM, Med-PaLM 2 [13] as a surrogate for a specialist rater to grade the predicted diagnoses against the ground truth diagnoses (more details in Section A.7). Our auto-evaluation on DDx accuracy showed a similar trend for AMIE and OSCE agents compared to the specialist ratings. Overall, auto-evaluation trends aligned with human ratings for both dialogue quality and diagnostic accuracy.