On-device machine learning has drawn increasing attention owing to the unique advantages [1]}, like real-time processing (without network latency regardless network connection), better privacy (without sending data to the cloud), and lower serving cost (without hosting massive cloud servers). On the other hand, the resource-constrained edge devices prevent many neural networks performing well in the on-device setting. In Natural Language Processing (NLP), on-device sequence-to-sequence (seq2seq) generation remains challenging, especially for the powerful Transformer [2]} under strict resource constraints in both computation and memory.
<FIGURE>