The test beds are benchmarks of knowledge-grounded dialogue generation, including Wizard of Wikipedia (Wizard) [1]}, CMU Document Grounded Conversations (CMU\(\_\) DoG) [2]}, and Topical-Chat (TC) [3]}, in which we discard the external knowledge passages. The remaining dialogues are basically deep discussions about specific topics between two participants, and thus are suitable probes. We choose DialoGPT [4]}, GPT-2\(_{finetune}\) , and DialoGPT\(_{finetune}\)  as the pre-trained language models for investigation, where GPT-2\(_{finetune}\)  and DialoGPT\(_{finetune}\)  refer to the OpenAI GPT-2 model [5]} and DialoGPT fine-tuned on the training data of the benchmarks respectively. Evaluation results on both automatic metrics and human judgment indicate that without the aid of external knowledge, the fine-tuned models are still capable of replying with proper and specific content based on the knowledge encoded in its parameters, though sometimes they may make mistakes on details.
