Since the year 2000, dialog systems (often called chatbots) have gone from mainly research demonstration systems to include various user-facing commercial offerings.
Dialogue systems fall into three broad categories [1]}:
conversational agents, question answering systems, and task-oriented systems (which are the topic of this paper).
Each category has corresponding dialog quality measurement strategies.
Conversational agents, which often receive the most attention in news articles when released,
are typically unstructured and open-domain, with no particular objective other than an engaging conversation.
Question answering systems can be evaluated by considering the accuracy of answers given.
Task-oriented systems, which are the focus of this article, typically have a rigid structure and a limited scope.
They are built to resolve the consumer's issue, answer questions, route to an appropriate representative, or guide the user through a task as efficiently as possible.
Like question answering systems, task-oriented systems have clear `failure' cases: just as a question answering system can fail to answer a question,
a task-oriented system can fail to complete a task.
