Recent successes of pretrained transformer-based language models, such as BERT [1]}, GPT(-2) [2]}, [3]}, Transformer-XL [4]}, XLNet [5]}, and ERNIE(2.0) [6]}, [7]}, have led to state-of-the-art performance on many natural language understanding (NLU) tasks including sentence classification, named entity recognition, sentence similarity, and question answering. The exceptional performance of transformer-based language models is due to their ability to capture long-term temporal dependencies in the input sequence. This attribute should be very beneficial to dialogue modeling, especially in multi-turn scenarios. Most of the existing neural dialogue response generation models are based on recurrent neural networks [8]}, [9]}, [10]}, [11]}, [12]}, [13]}, [14]}, [15]}, [16]}, [17]}, [18]}.
