Hyperparameters We generally follow the hyperparameters for a transformer base (Vaswani et al., 2017;<cite> Ghazvininejad et al., 2019)</cite> : 6 layers for both the encoder and decoder, 8 attention heads, 512 model dimensions, and 2048 hidden dimensions.