As in-domain dialogue data are usually scarce, we use a two-stage pretraining strategy to enhance the noisy channel model. Although the effectiveness of pretraining with Reddit data has been validated for open-domain dialogue [1]}, [2]}, [3]}, relatively little work has applied such data to task-oriented dialogue.One exception is Henderson et al. henderson2019training, who use Reddit data to improve response retrieval and selection. We focus on response generation in this work. In the first stage, we explore Reddit pretraining (where the Reddit data is pre-processed into \((\mathcal {C}, \mathcal {R})\) , i.e. context-response, pairs as described below). In the second stage, we use two task-oriented dialogue datasets, Taskmasterhttps://cutt.ly/xkuUHUa [4]} and Schema-Guided Dialoguehttps://cutt.ly/QkuUZUu [5]}, to specialize the Reddit-pretrained models. Since the Reddit data consists of open-domain-style dialogues (where belief states and dialogue acts are missing), pretraining on these datasets can familiarize the models with the sequence-to-sequence representation of task-oriented dialogue. Three models, a context-to-response model, a response-to-context model and a response language model, are pretrained to initialize the direct model, the channel model and the source model, respectively.
<TABLE>