There have been a few studies on <cite>transformers</cite> for end-to-end speech recognition, particularly for sequence-to-sequence with attention model [10, 11, 12] , as well as transducer [13] and CTC models [14] .