Experiment DetailsFor reasoning, NLU, and knowledge question answering tasks, we adopt accuracy as the metric. As for the NLG tasks, we adopt ROUGE-L as the metric. Furthermore, for the concrete numbers of different layers, we tune hyperparameters by XQuAD in Chinese. Details are explained in Appendix B.