Since the decoded sequence is the output for previous timestep, the query at position i can only observe the keys being the tokens that are decoded with position < i. For convenience, let us define S 1 as the set returned by original <cite>Transformer</cite> <cite>(Vaswani et al., 2017 )</cite> from M (x q , S x k ), which we will use it later.