Different from the architectures of convolutional neural network (CNNs) and recurrent neural networks (RNNs) language modeling, the Transformer<cite> [34]</cite> and its variants [7, 12, 10] achieve excellent results in language modeling processing.