ii) Short-range Model Architecture: Earlier conversation models are single-turn Seq2Seq architectures [1]}, [2]}, [3]} that fails to capture long-term temporal dependencies across conversation turns. Such models tend to fail in multi-turn scenarios [4]}, generating repetitive responses that are dull and generic. The use of multi-turn Seq2Seq models, such as the hierarchical recurrent encoder decoder (HRED) architecture, tried to address this problem [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}. The recurrent architecture, however, due to the gradient vanishing problem with backpropagation through time, limits the maximum number of turns and the number of word tokens in each turn that are used during training.
