During pre-training, Label-aware masked language model (LA-MLM) and next sentence prediction (NSP) are adopted as the pre-training tasks where the setting of NSP is identical to the one proposed by<cite> Devlin et al. (2019)</cite> .