[1]} list 21 dimensions across 20 publications that human evaluators have used to measure the quality of open-domain dialog systems.
While these dimensions are not used directly, they partially inspire our taxonomy of failure states (Table , leftmost column).
For instance the `Doesn’t Understand' failure state is inspired by Coherence [2]}, [3]}, Correctness [4]}, [5]}, Relevance [6]}, [7]}, [8]}, Logic [9]} and Sensibleness [10]}.
Other elements of our taxonomy are informed by [11]} which provides a set of best practices when developing dialog systems for messaging based bots. We separated `misunderstanding' into `Input Rejected', `Ignores Consumer' and `Does Not Understand' categories, because each of these require different
actions that can mitigate the understanding issue. Such separation of failure states highlights the design-for-actionability of the ACQI taxonomy.
