Insertion Deletion Transformer
We propose the Insertion Deletion Transformer, a novel transformer based neural architecture and training method for sequence generation. The model consists of two phases that are executed iteratively, 1) an insertion phase and 2) a deletion phase. The insertion phase parameterizes a distribution of insertions on the current output hypothesis, while the deletion phase parameterizes a distribution of deletions over the current output hypothesis. The training method is a principled and simple algorithm, where the deletion model obtains its signal directly on policy from the insertion model output. We demonstrate the effectiveness of our Insertion Deletion Transformer on synthetic translation tasks, obtaining significant BLEU score improvement over an insertion only model.
ArXiv