For EdgeFormer, we parameterize its encoder using as many parameters as possible: except a small number of parameters (\(d^2/2\) ) for all lightweight FFNs in the decoder, we use almost all parameters in our budget to parameterize the encoder. In this way, our parameterization budget allows 4 groups of attention parameters (\(4\times 4d^2\) ), 2 groups of encoder FFN parameters (\(2\times 8d^2\) ) and 1 group for decoder lightweight FFNs (i.e., \( d^2/2\) ). We do not assign parameters exclusively for attention modules in the decoder; instead, we let them reuse (i.e., share) parameters with the attention modules in the encoder since attention modules in both the encoder and decoder work in the same mechanism and can be effectively shared [1]}. Thanks to the interleaved decoder architecture that makes the structure of encoder and decoder layers consistent, we let the self-attention module in a decoder layer share parameters with its corresponding odd layer in the encoder, and let its cross-attention module share with the corresponding even layer in the encoder, inspired by [2]}:
\(W^{[Q,K,V,O]}_{d_{j}}=W^{[Q,K,V,O]}_{e_{2j-1}} ~~~ (1 \le j \le 2)\) 
\(W^{[\mathcal {Q},\mathcal {K},\mathcal {V},\mathcal {O}]}_{d_{j}}=W^{[Q,K,V,O]}_{e_{2j}} ~~~(1 \le j \le 2)\) 
