Similar to CodeT5  [1]}, we use the RoBERTa [2]} tokenizer to tokenize all sequences (input, edit plan, target). More concretely, the tokenizer splits words in the sequence into subwords, constituting the tokens that are used by the model.
Moreover, we remove input sequences that are shorter than 3 tokens and longer than 512 tokens after tokenization.
Statistics of the pretraining dataset are presented in Table .
