On-device seq2seq generation is a research area that has been much less explored than on-device CV. Besides the general techniques like quantization, pruning and knowledge distillation, there are some research branches closely related to this paper that focus on seq2seq learning. Among them is parameter-efficient Transformer modeling. In this branch, Universal Transformer [1]} uses cross-layer sharing method, which is the most popular way for saving parameters, although its original motivation is not saving parameters. [2]} extends Universal Transformer by studying different ways for layer sharing, and [3]} proposes to free the first and last encoder layer and widen the intermediate layers for better performance. However, both the approaches consider parameter-efficiency only without caring decoding latency that becomes much worse.
