**Benchmarking Auto-evaluation.** While auto-evaluation was significantly better than random guessing at aligning with specialist preferences, it was unclear if the resulting performance was sufficient. To test this, we had 139 dialogue pairs each rated by a second specialist, and we computed the rank-order agreement of the two specialists on this subset (see figure A.18). We observed that auto-evaluation was about as accurate as the alternative specialists in predicting the first specialist's rank-order, suggesting that it is useful to leverage auto-evaluation for these criteria.