It seems that pre-training decoders has a smaller effect on the final performance than pre-training encoders <cite>(Lample and Conneau, 2019)</cite> , one reason for which could be that the encoder-to-decoder attention is not pre-trained.