Although there are a number of metrics for nlp systems [1]}, [2]}, [3]} different metrics may be suitable for different systems, depending on the characteristics of the system.
For example, the goals of task-based systems are different from those of open-domain conversational systems, so they may not use the same evaluation metrics.
Human evaluation is the gold standard in the evaluation of open-domain conversational ai, though it is subjective [4]}.
It is both time-intensive and laborious.
As a result of this, automatic metrics serve as proxies for estimating performance though they may not correlate very well with human evaluation [2]}, [6]}, [7]}.
For example, ir systems may use F1, precision, and recall [1]}.
Furthermore, metrics used in nlg tasks, like mt, such as the bleu or ROUGE, are sometimes used to evaluate conversational systems [4]} but they are also discouraged because they do not correlate well with human judgment [10]}, [11]}.
They do not take syntactic or lexical variation into consideration [3]}.
Dependency-based evaluation metrics, however, allow for such variation in evaluation.
Perplexity is commonly used for evaluation and has been shown to correlate with a human evaluation metric called Sensibleness and Specificity Average (SSA) [13]}.
It measures how well a model predicts the data of the test set, thereby estimating how accurately it expects the words that will be said next [13]}.
It corresponds to the effective size of the vocabulary [1]} and smaller values show that a model fits the data better.
Very low perplexity, however, has been shown to suggest such text may have low diversity and unnecessary repetition [16]}.
