DLGNet adopts GPT-2's autoregressive transformer architecture [1]} using only the decoder part of the original transformer architecture [2]} since there is no need for a separate encoder network.
Autoregressive transformer models use multiple layers of masked multi-head self-attention to map a sequence of input tokens to a sequence of output tokens (i.e., the input sequence token shifted one position to the right).
During inference, at each step, the model is autoregressive, consuming the previously generated token as additional input when generating the next.
There are some basic conceptual differences between autoregressive architectures based on transformers and those based on recurrent neural networks (RNNs).
For instance, while the output of an RNN layer depends on only the immediate previous output, a transformer layer output consists of attention over all previous outputs.
Due to this lack of ordering in transformer architectures, the position representation is usually passed along with the input tokens into the model [2]}.
