**Rank-order Agreement.** We evaluated our auto-evaluation method by quantifying its agreement with the specialist rankings of the OSCE dialogues. We limited our analysis to the 149 dialogue pairs in the study. Thus, each pair consisted of a AMIE conversation and a PCP conversation with the same patient actor, and rated by the same specialist. For a pair of two dialogues, the three possibilities were: the first one was rated better than the second one, they were equally rated, or the first one was rated worse than the second one. We defined the rank-order agreement as the proportion of dialogue pairs for which the specialist ranking was preserved by the auto-evaluation ratings. For example, we counted it as correct when our auto-evaluation rated AMIE's dialogue as better than the PCP's dialogue if specialists also rated AMIE's dialogue as better, regardless of the exact scores each method assigned.