As the memory cost of self-attention is in the order of O(T 2 ), where T is the length of the acoustic sequence, lower frame rate would significantly cut down the memory cost, and enable the training of much deeper <cite>transformer models</cite> that will studied in our future work.