Rothe et al. [1]} demonstrated the effectiveness of pre-trained checkpoints for sequence generation.
We use pre-trained bert-base-uncased [2]} model as the encoder and the decoder since it showed good performance for translation task [1]}. Since the BERT is inherently an encoder model while using it as the decoder, additional cross attention weights are introduced and these are initialized randomly. Therefore, the encoder and the decoder both have 12 layers. The model is fine-tuned for 3 epochs using AdamW optimizer with a learning rate of 5e-6 and batch size of 32.
