The Capacity for Moral Self-Correction in Large
Language Models
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao
The Capacity for Moral Self-Correction in Large Language Models
We test the hypothesis that language models trained with reinforcement
learning from human feedback (RLHF) have the capability to "morally
self-correct" -- to avoid producing harmful outputs -- if instructed to do so.
We find strong evidence in support of this hypothesis across three different
experiments, each of which reveal different facets of moral self-correction. We
find that the capability for moral self-correction emerges at 22B model
parameters, and typically improves with increasing model size and RLHF
training. We believe that at this level of scale, language models obtain two
capabilities that they can use for moral self-correction: (1) they can follow
instructions and (2) they can learn complex normative concepts of harm like
stereotyping, bias, and discrimination. As such, they can follow instructions
to avoid certain kinds of morally harmful outputs. We believe our results are
cause for cautious optimism regarding the ability to train language models to
abide by ethical principles.
Deep Ganguli Amanda Askell Nicholas Schiefer Thomas I. Liao Kamilė Lukošiūtė Anna Chen Anna Goldie Azalia Mirhoseini Catherine Olsson Danny Hernandez Dawn Drain Dustin Li Eli Tran-Johnson Ethan Perez Jackson Kernion Jamie Kerr Jared Mueller Joshua Landau Kamal Ndousse Karina Nguyen Liane Lovitt Michael Sellitto Nelson Elhage Noemi Mercado Nova DasSarma Oliver Rausch Robert Lasenby Robin Larson Sam Ringer Sandipan Kundu Saurav Kadavath Scott Johnston Shauna Kravec Sheer El Showk Tamera Lanham Timothy Telleen-Lawton Tom Henighan Tristan Hume Yuntao Bai Zac Hatfield-Dodds Ben Mann Dario Amodei Nicholas Joseph Sam McCandlish Tom Brown Christopher Olah Jack Clark Samuel R. Bowman Jared Kaplan