We implemented two separate Transformer models for neural deletion and split component (§REF ) and paraphrase generation (§REF ) using the Fairseqhttps://github.com/pytorch/fairseq toolkit. Both the encoder and decoder follow BERT\(_{base}\)https://github.com/google-research/bert architecture, while the encoder is also initialized with BERT\(_{base}\)  checkpoint. For neural deletion and split component, we used a beam search of width 10 to generate candidates. The copy attention mechanism is a feedforward network containing 3 hidden layers, 1000 nodes in each layer with tanh activation, and a single linear output node with sigmoid activation. We used Adam optimizer [1]} with a learning rate of 0.0001, linear learning rate warmup of 40k steps, and 100k training steps. We used a batch size of 64. We used BERT WordPiece tokenizer. During inference, we constrained the beam-search to not repeat trigrams and emitted sentences that avoided aggressive deletion (compression ratio \(\in [0.9, 1.2]\) . We chose the best checkpoint based on the SARI score [2]} on the dev set. We saved a checkpoint after every epoch. We did not perform any hyperparameter search and directly used the hyperparameters of the BERT-initialized Transformer described in [3]}. The model takes 10 hours to train on 1 NVIDIA GeForce GPU.
