Preprint. Under review.
GENERATING SEQUENCES BY
LEARNING TO [SELF-]CORRECT
Sean Welleck1,3,*Ximing Lu1,*
Peter West3,yFaeze Brah
Generating Sequences by Learning to Self-Correct
Sequence generation applications require satisfying semantic constraints,
such as ensuring that programs are correct, using certain keywords, or avoiding
undesirable content. Language models, whether fine-tuned or prompted with
few-shot demonstrations, frequently violate these constraints, and lack a
mechanism to iteratively revise their outputs. Moreover, some powerful language
models are of extreme scale or inaccessible, making it inefficient, if not
infeasible, to update their parameters for task-specific adaptation. We present
Self-Correction, an approach that decouples an imperfect base generator (an
off-the-shelf language model or supervised sequence-to-sequence model) from a
separate corrector that learns to iteratively correct imperfect generations. To
train the corrector, we propose an online training procedure that can use
either scalar or natural language feedback on intermediate imperfect
generations. We show that Self-Correction improves upon the base generator in
three diverse generation tasks - mathematical program synthesis,
lexically-constrained generation, and toxicity control - even when the
corrector is much smaller than the base generator.
Sean Welleck Ximing Lu Peter West Faeze Brahman Tianxiao Shen Daniel Khashabi Yejin Choi