Since human labor is expensive, we only compare the pre-trained language models with DRD on Wizard. For each of Test Seen and Test Unseen, we randomly sample 500 contexts. Responses from different models for these contexts are pooled, randomly shuffled, and presented to 4 well-educated native speakers as annotators. The annotators judge the quality of each response from five aspects: fluency, coherence regarding the contexts, if the response carries knowledge (knowledge existence), if the knowledge is relevant with the contexts (knowledge relevance), and if the knowledge is correct (knowledge correctness). On each aspect, a binary label (0/1) is assigned to a response by an annotator, and in total each response receives 4 labels per aspect. Annotators are encouraged to actively use search engines when they feel difficult to make a decision. Note that if an annotator judges there is no knowledge in a response, then by default he/she will also assign 0s to the response on knowledge relevance and knowledge correctness. The agreement among the annotators is measured by Fleiss’ kappa [1]}.
