This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing masked language modeling (MLM)  with replaced token detection (RTD) , a more sample-efficient pre-training task.
Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance, because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the “tug-of-war” dynamics.
We thus propose a newgradient-disentangled embedding sharingmethod that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model.
We have pre-trained DeBERTaV3  using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU)  tasks.
Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3  Large model achieves a 91.37% average score, which is 1.37% higher than DeBERTa and 1.91% higher than ELECTRA, setting a new state-of-the-art (SOTA)  among the models with a similar structure.
Furthermore, we have pre-trained a multilingual model mDeBERTaV3  and
observed a larger improvement over strong baselines compared to English models.
For example, the mDeBERTaV3  Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. Our models and code are publicly available athttps://github.com/microsoft/DeBERTa.