As mentioned earlier, we can take advantage of recent pre-trained Transformer encoders for the document encoding part as in<cite> Liu and Lapata (2019)</cite> . However,<cite> Liu and Lapata (2019)</cite> leave the decoder randomly initialized.