ACUTE -EVAL : Improved dialogue evaluation with optimized questions and
multi-turn comparisons
Margaret Li
Facebook AI Research

ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
While dialogue remains an important end-goal of natural language research,
the difficulty of evaluation is an oft-quoted reason why it remains troublesome
to make real progress towards its solution. Evaluation difficulties are
actually two-fold: not only do automatic metrics not correlate well with human
judgments, but also human judgments themselves are in fact difficult to
measure. The two most used human judgment tests, single-turn pairwise
evaluation and multi-turn Likert scores, both have serious flaws as we discuss
in this work.
  We instead provide a novel procedure involving comparing two full dialogues,
where a human judge is asked to pay attention to only one speaker within each,
and make a pairwise judgment. The questions themselves are optimized to
maximize the robustness of judgments across different annotators, resulting in
better tests. We also show how these tests work in self-play model chat setups,
resulting in faster, cheaper tests. We hope these tests become the de facto
standard, and will release open-source code to that end.
Margaret Li Jason Weston Stephen Roller