<cite>Transformer</cite> <cite>[8]</cite> , which relies solely on self-attention to capture the temporal correlations in sequential signals, is a new type of neural network structure for sequence modeling, which has achieved excellent results in machine translation <cite>[8]</cite> , language modeling [9] , as well as end-to-end speech recognition [10, 11] .