Sparse Transformer (Child et al., 2019) showed that although having much fewer elements in M (x q , S x k ), if the elements are carefully chosen, the attention can still reach the same performance as Transformer-XL<cite> (Dai et al., 2019)</cite> .