Since our method can adapt to all the BERTstyle pre-training models, we used vanilla BERT<cite> (Devlin et al., 2019)</cite> as the base framework to construct Transformer blocks in this paper and leave the exploration of other models like RoBERTa as future work.