We use the standard conformer block[1]}, which is composed of four modules stacked together, i.e, a feed-forward module, a self-attention module, a convolution module, and a second feed-forward module, the two Feed Forward modules sandwiching the Multi-Headed Self-Attention module and the Convolution module. Mathematically, this means, for input \(\mathbf {x}_{i}\) 
to a Conformer block \(i\) , the output \(\mathbf {y}_{i}\)  of the block is:
\(\widetilde{\mathbf {x}}_{i} = LN(\mathbf {x}_{i} + \frac{1}{2}FFN(\mathbf {x}_{i}))\) 
\({\mathbf {x}_{i}}\\^{\prime } = LN(\widetilde{\mathbf {x}}_{i} + MHSA(\widetilde{\mathbf {x}}_{i}))\) 
\({\mathbf {x}_{i}}\\^{\prime }\\^{\prime } = LN({\mathbf {x}_{i}}\\^{\prime } + Conv({\mathbf {x}_{i}}\\^{\prime }))\) 
\({\mathbf {y}_{i}} = LN({\mathbf {x}_{i}}\\^{\prime }\\^{\prime } + \frac{1}{2}FFN({\mathbf {x}_{i}}\\^{\prime }\\^{\prime }))\) 
