Experimental results on both WMT14 En\(\Rightarrow \) De and En\(\Rightarrow \) Fr translation tasks are listed in Table REF .
As seen, exploiting deep sentential context representation consistently improves translation performance across language pairs and model architectures, demonstrating the necessity and effectiveness of modeling sentential context for NMT. Among them, Transformer-Base with deep sentential context achieves comparable performance with the vanilla Transformer-Big, with only less than half of the parameters (114.4M vs. 264.1M, not shown in the table).
Furthermore, Deep (Tam) consistently outperforms Deep (RNN) in the Transformer-Big configuration. One possible reason is that the big models benefit more from the improved gradient flow with the transparent attention [1]}.
<TABLE>