Transformer [1]} was first brought to machine translation with the architecture of encoder-decoder.
Later works [2]}, [3]} show that it remains effective, even when the encoder or the decoder is removed, reducing nearly half of the parameters.
Under the paradigm of pre-training plus fine-tuning, Transformer's effectiveness has been confirmed on a wide range of tasks, including both natural language understanding and generation [4]}, [3]}, [6]}.
Particularly, it is able to perform novel tasks, e.g., arithmetic, after scaling up both the model and the training data [7]}, [8]}.
However, it may not be friendly to researchers who do not possess large amounts of computing resources.
Instead, our work explores small unpretrained models, as they are computationally cheaper and more flexible when being adapted to new applications, e.g., personalized generation.
