While whole sequence-level self-attention has been applied in sequence-to-sequence models [10] , hybrid model is different in the sense that it is required to maintain strict frame-level alignments before performing predictions, which may be challenging for a <cite>transformer</cite> with multiple layers of self-attention as it may reorder the sequence.