It has become common in recent years to pretrain dialogue models on large text data, either general text [1]}, [2]}, [3]} or dialogue-structured data [4]}, [5]}, such as tweets and Reddit posts. We utilise a similar strategy with Reddit data and find that the benefits of pretraining to the noisy channel model are similar to those for the direct model.
Further, we evaluate transfer across task-oriented dialogue datasets by implementing a second pretraining stage using Taskmaster [6]} and Schema-Guided Dialogue [7]} as training data, before fine-tuning on our final tasks.
