Intuitively, edit-based generation involves a different mentality that
more frequently refers back to the input sequence, and can often be
characterized by localized operations (e.g., insertion, deletion,
substitution). This paper's first contribution is to formulate a novel pretraining objective that
explicitly models edits. It guides a model in learning to discern edit locations in the input
sequence and reason about the necessary edit
operations. Our approach is inspired by content planning in natural language generation where a skeleton of key elements are first generated and used to guide more accurate and precise generation of full text [1]}, [2]}, [3]}, [4]}.
Specifically, we first generate an edit plan that explicitly details the edit
operations before generating the target sequence as a result of the edits.
This effectively allows the decoder to condition on the edit plan during decoding.
Using this objective, we develop CoditT5, a large language model for software-related edit tasks that is pretrained on more than 5.9 million open-source programming language functions and 1.6 million natural language comments from the CodeSearchNet [5]} training data.
