Sparse <cite>Transformer</cite> (Child et al., 2019) showed that although having much fewer elements in M (x q , S x k ), if the elements are carefully chosen, the attention can still reach the same performance as <cite>Transformer</cite>-XL (Dai et al., 2019) .