To measure the performance of the dialogue policy, we follow the approach of <cite>DeVault et al. (2011)</cite> , which counts an automatically produced system SA as correct if that SA was chosen by at least one referee for that dialogue turn in the data set.