The original Transformer is a sequence-to-sequence model designed in an encoder-decoder type of configuration that takes as input a sequence of words from the source language and then generates the translation in the target language [1]}, [2]}. Given that the length of the two sequences and the vocabulary size are not necessarily the same, the model has to learn to encode the source sequence into a fixed-length representation that it can then decode to generate the target sequence in an auto-regressive fashion [3]}. This auto-regressive property comes with a constraint of requiring information to propagate back to the beginning of the sequence during the generation of the translated sequences. The same constraint holds for the time-series analysis.
