Two methods for human evaluation of open-domain conversational ai are observer and participant evaluation [1]}.
Observer evaluation involves reading and scoring a transcript of human-chatbot conversation while participant evaluation interacts directly with the conversational ai [1]}.
In the process, the system may be evaluated for different qualities, such as humanness (or human-likeness), fluency, making sense, engagingness, interestingness, avoiding repetition, and more.
The Likert scale is usually provided for grading these various qualities.
The others are comparison of diversity and how fitting responses are to the given contexts.
Human evaluation is usually modeled to resemble the Turing test (or the imitation game).
