Figure A.18: Rank-order agreement to specialist ratings of 139 dialogue pairs (excluding cases without multiple specialist ratings) for alternative specialists compared to the self-CoT auto-evaluation technique. Auto-evaluation agreement to the first specialist is comparable to inter-specialist agreement. The black dashed line shows the rank-order agreement one would get with a random ranking of the AMIE and PCP dialogues, while the green dashed line shows the rank-order agreement with a strategy of randomly guessing according to the distribution of specialist preferences for each criteria.