Large Pretrained Models for Code.
Motivated by the success of large pretrained models for many NLP
tasks, domain-specific models that are pretrained on source code and
technical text have emerged, including
CodeBERT [1]},
GraphCodeBERT [2]},
CodeGPT-2 [3]}, CodeT5 [4]},
PLBART [5]}, PyMT5 [6]},
SYNCOBERT [7]}, SPT-Code [8]},
Codex [9]} and UniXcoder [10]}.
Similar to our approach, GraphCodeBERT, CodeT5 and UniXcoder also designed specialized pretraining objectives driven by their targeted tasks.
While they demonstrate impressive performance on various tasks, none of them are fundamentally well-suited for edit tasks.
In this work, we develop CoditT5 with a novel pretraining objective for generating edit sequences, which can complement the generation model such as CodeT5 for edit tasks.
