In recent years, transformer-based models [1]} have been ubiquitous in various natural language processing (NLP) applications, including machine translation[2]}, question answering and document analysis[3]}. Compared with traditional recurrent neural network (RNN) and long short-term memory (LSTM)[4]}, transformer-based model uses multi-head self-attention mechanism to encode and decode the information of input sentence to achieve better representative ability, and the Bidirectional Encoder Representations from Transformers (BERT)[5]} is the state-of-the-art transformer-based model.
