In this work, we investigate if a large scale pre-trained language model can instead serve as a knowledge base in open domain dialogue generation. The work is motivated by two lines of research emerging recently: (1) interestingly, some recent studies indicate that pre-trained language models have packed enough knowledge in their parameters, and thus they can do a good job in question-answering tasks without the need of access to external knowledge [1]}, [2]}. Thus, we are curious if similar results can be achieved in open domain dialogue generation. If the answer is “yes”, then we can get rid of the dependency on external knowledge sources, and obtain a simpler and more flexible architecture with a better generalization ability inherited from pre-training with massive text corpus [3]}; and (2) pre-training techniques have exhibited compelling performance on the task of open domain dialogue generation [4]}, [5]}. Particularly, a recent paper [4]} has demonstrated with examples that a pre-trained generation model can reply with commonsense knowledge. Therefore, it is interesting to further explore to what extend a pre-trained language model can keep conversation smooth, knowledgeable, and reasonable with in-depth analysis.
