\(\bullet\)_LayerNorm._ In the early research, BatchNorm [265] is a commonly used normalization method. However, it is difficult to deal with sequence data of variable lengths and small-batch data. Thus, LayerNorm [256] is introduced to conduct layerwise normalization. Specifically, the mean and variance over all activations per layer are calculated to recenter and re-scale the activations.