Another line of work that carries out the idea of learning edits is
copying mechanism, including copying individual
tokens [1]}, [2]} and
spans [3]}, [4]}, which
helps the model to “keep” unchanged tokens and focus on generating
the edited part.
[5]} built a T5-based model to update the existing articles based on the given new evidence. The model is trained to output a copy token instead of the copied sentence and a special reference token before the updated text which identifies the evidence to support the update.
[6]} trained the model to emit
pointers that indicate the positions for editions and new tokens
to be inserted at the same time.
Similarly, [7]}, [8]} augmented the transformer-based decoder with pointers to the input graph representation of the code which specify the input locations to edit.
Although related, it is orthogonal to our work of
learning edits with pretraining.
