In this paper, we propose DLGNet, a transformer-based model for multi-turn dialogue modeling that addresses some of the highlighted problems above. The use of a transformer architecture allows DLGNet to capture long-term temporal dependencies in the dialogue data better than the existing RNN-based architectures [1]}. However, applying a vanilla Seq2Seq transformer [1]} for dialogue modeling does not work well because of the semantic redundancy in dialogue data. To overcome this, DLGNet models the joint distribution of the context and response instead of the conditional distribution of the response given the context, usually employed in Seq2Seq frameworks [3]}, [4]}, [5]}, [1]}. DLGNet also addresses the syntactic redundancy in dialogue data by appending random paddings before and after the input data. This helps to break down the learning barrier from the concave entropy profile of human conversation data, as shown in Fig. REF . The flattening of the entropy profile also provides regularization during training, and reduces even the extent of the exposure bias problem. Finally, to avoid the out-of-vocabulary problem, DLGNet uses byte pair encoding (BPE) similar to GPT-2 [7]} to provide 100% coverage for any Unicode input and output texts.
Given all these proposed changes, we train DLGNet models using only the maximum likelihood objective.
DLGNet models, despite being trained with only the maximum likelihood objective, demonstrate state-of-the-art performance on the Movie and Ubuntu datasets, as measured in terms of BLEU, ROUGE, and distinct n-gram scores.
<TABLE>