WMT14.
We use the fixed sinusoidal position embeddings by [1]}.
For both EN-DE and EN-FR experiments,
we train the models using the Adam (with \(\beta _1=0.1\) , \(\beta _2=0.98\) , and \(\epsilon =10^{-9}\) ) optimizer
for up to 350K gradient steps.
We use a batch size of 1,024 instances for EN-DE,
while 4,096 for the much larger EN-FR dataset.
The learning rate follows that by [1]}.
Early stopping is applied based on development set BLEU.
No \(\ell _2\)  regularization or gradient clipping is used.
All models are trained using 16 TPU v3 accelerators,
and tested using a single TPU v2 accelerator.
Following standard practice, we average 10 most recent checkpoints at test time.
We evaluate the models using SacreBLEU [3]}.https://github.com/mjpost/sacrebleu
A beam search with beam size 4 and length penalty 0.6 is used.
Other hyperparameters are summarized in Table REF .
<TABLE>