We conducted experiments on WMT14 En\(\Rightarrow \) De and En\(\Rightarrow \) Fr benchmarks, which contain 4.5M and 35.5M sentence pairs respectively.
We reported experimental results with case-sensitive 4-gram BLEU score. We used byte-pair encoding (BPE) [1]} with 32K merge operations to alleviate the out-of-vocabulary problem.
We implemented the proposed approaches on top of Transformer model [2]}. We followed Vaswani:2017:NIPS to set the model configurations, and reproduced their reported results.
We tested both Base and Big models, which differ at the layer size (512 vs. 1024) and the number of attention heads (8 vs. 16).
