We compare the models using two model size settings, summarized in
Table REF .
We use the fixed sinusoidal position embeddings by [1]}.
All models are trained for up to 150K gradient steps
using the Adam optimizer [2]}.
No \(\ell _2\) -regularization is used.
We apply early stopping based on development set perplexity.
All models are trained using 16 TPU v3 accelerators,
and tested using a single TPU v2 accelerator.
<TABLE>