@inproceedings{
anil2024manyshot,
title={Many-shot Jailbreaking},
author={Cem Anil and Esin DURMUS and Nina Rimsky and Mrinank Sharma and Joe Benton and Sandipan Kundu and Joshua Batson and Meg Tong and Jesse Mu and Daniel J Ford and Francesco Mosconi and Rajashree Agrawal and Rylan Schaeffer and Naomi Bashkansky and Samuel Svenningsen and Mike Lambert and Ansh Radhakrishnan and Carson Denison and Evan J Hubinger and Yuntao Bai and Trenton Bricken and Timothy Maxwell and Nicholas Schiefer and James Sully and Alex Tamkin and Tamera Lanham and Karina Nguyen and Tomasz Korbak and Jared Kaplan and Deep Ganguli and Samuel R. Bowman and Ethan Perez and Roger Baker Grosse and David Duvenaud},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=cw5mgd71jW}
}
We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This attack is newly feasible with the larger context windows recently deployed by language model providers like Google DeepMind, OpenAI and Anthropic. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.
Many-shot Jailbreaking
Cem Anil Esin DURMUS Nina Rimsky Mrinank Sharma Joe Benton Sandipan Kundu Joshua Batson Meg Tong Jesse Mu Daniel J Ford Francesco Mosconi Rajashree Agrawal Rylan Schaeffer Naomi Bashkansky Samuel Svenningsen Mike Lambert Ansh Radhakrishnan Carson Denison Evan J Hubinger Yuntao Bai Trenton Bricken Timothy Maxwell Nicholas Schiefer James Sully Alex Tamkin Tamera Lanham Karina Nguyen Tomasz Korbak Jared Kaplan Deep Ganguli Samuel R. Bowman Ethan Perez Roger Baker Grosse David Duvenaud