We select Universal Transformer which is the most popular parameter-efficient Transformer as our main baseline for fair comparison. By default, we apply Seq-KD [1]} to train models, since there is no reason for training smaller models in practice without Seq-KD that is effective to improve results with no harm. The full-parameterized 6+6 Transformer-big (\(d=1,024\) ) model [2]}, [3]} is used as the teacher. For inference, we use beam search (beam=5) by default.
<TABLE>