In addition to work improving parameter efficiency by weight sharing, there is research that studies lightweight model architecture for seq2seq learning where early work [1]}, [2]} mainly focuses on CNNs, while recent efforts have tended to switch to attention-based models such as [3]}. Also, low-rank factorization has been studied intensively to make the model tiny [4]}, [5]}. Among previous studies, the work of [6]} is the most related to ours, which studies seq2seq generation in a mobile setting. However, it sets the computational constraint for on-device seq2seq to be the same with the CV tasks, which is too strict and unnecessary, as discussed in Section REF . As a result, either models that they studied are too large and cost massive memory or fail to achieve desirable performance despite high inference speed. In contrast, our work carefully evaluates the bottleneck constraints, and proposes appropriate models with parameterization and layer adaptation innovations, largely improving the results for practical on-device seq2seq generation.
