For performance-wise comparisons, <cite>Transformer</cite>-XL (Dai et al., 2019) showed that, the additional memories in M (x q , S x k ) are able to capture longer-term dependency than the original <cite>Transformer</cite> <cite>(Vaswani et al., 2017)</cite> and hence results in better performance.