(iv) Decoder Self-Attention in Transformer-XL<cite> (Dai et al., 2019)</cite> : For each query x q in the decoded sequence, M (x q , S x k ) returns a set containing S 1 and additional memories (M (x q , S x k ) = S 1 + S mem , M (x q , S x k ) ⊃ S 1 ).