Pretraining models for dialogue: Recent work has applied pretraining [1]}, [2]}, [3]} to dialogue. For open-domain dialogue, DialoGPT [4]} and CGRG [5]} extend GPT-2 [3]} for response generation. PLATO [7]} and PLATO-2 [8]} pretrain a latent variable model with social media data for diversified response generation. Meena [9]} collects a large-scale social media corpus for pretraining and proposes a metric named sensibleness and specificity average for evaluation. Roller et al. roller2020recipes study various strategies for building an open-domain chatbot with Reddit for pretraining. For task-oriented dialogue, ToD-BERT [10]} fine-tunes BERT [2]} for four tasks, including intention detection, belief tracking, dialogue act prediction, and response selection. SC-GPT [12]} fine-tunes GPT-2 for few-shot response generation with given dialogue acts. Ham et al. ham2020end fine-tune GPT-2 for belief tracking and context-to-response generation. SimpleTOD [13]} proposes a method to serialize dialogue beliefs and acts into text spans and fine-tunes GPT-2 for end-to-end dialogue modeling. SOLOIST [14]} uses a series of task-oriented dialogue datasets to further pretrain GPT-2 before fine-tuning it on final tasks for evaluation. Unlike these BERT- or GPT-initialized task-oriented dialogue models, which are essentially pretrained with general text, such as Wikipedia and BookCorpus, we use a Reddit dump to pretrain the models to learn from open-domain dialogues.
