For Transformer, PETER and PETER+, we set the embedding size \(d\)  to 512 and the dimension of feed-forward network to 2,048, following [1]}, but the number of layers \(L\)  and attention heads \(H\)  are both 2.
For our models PETER and PETER+, we set the regularization weights \(\lambda _e\) , \(\lambda _c\)  and \(\lambda _r\)  to 1.0, 1.0 and 0.1, respectively.
We optimize the model via stochastic gradient descent [2]}, and apply gradient clipping [3]} with a threshold of 1.0.
The batch size is set to 128, and the learning rate 1.0.
At each epoch, we save the model if it achieves the lowest loss on the validation set, but when there is no improvement, we decrease the learning rate by a factor of 0.25.
When the latter happens for 5 times, we stop training and load the saved model for prediction.
