Table REF  reports the evaluation results. Obviously, DialoGPT is much worse than the baselines in terms of all metrics over all the three datasets, due to the gap between conversations in Reddit (at least those for training) and conversations in the probes. The results indicate that DialoGPT under a zero-shot setting is still not capable of going deep in discussions about a specific topic, even though its responses carry some content case-by-case as shown in [1]}. However, this does not mean that there is “no knowledge” in the pre-trained language models, as after fine-tuning on the training sets of the probes, DialogGPT is comparable with
the baselines on most of the metrics. Note that we follow [2]} and exploit a vocabulary with \(60,000\)  words for the baselines. Although the difference in vocabulary size is not significant, PPL is still not directly comparable between the pre-trained models and the baselines. In terms of F1, DialogGPT\(_{finetune}\)  is comparable with the baselines on Wizard, and outperforms the baselines on the other data, indicating that the model can leverage the knowledge in its parameters and reply with relevant and specific content along with topic of the dialogue contexts, at least judged from unigram overlap with human responses. The advantage of DialogGPT\(_{finetune}\)  over the baselines is even bigger on CMU\(\_\) DoG, indicating that pre-training is more crucial when training data is small in size. GPT-2\(_{finetune}\)  is worse than DialogGPT\(_{finetune}\) , probably due to the small training size of the probes.
