For performance-wise comparisons, Transformer-XL<cite> (Dai et al., 2019)</cite> showed that, the additional memories in M (x q , S x k ) are able to capture longer-term dependency than the original Transformer (Vaswani et al., 2017) and hence results in better performance.