@inproceedings{
kotha2024understanding,
title={Understanding Catastrophic Forgetting in Language Models via Implicit Inference},
author={Suhas Kotha and Jacob Springer and Aditi Raghunathan},
booktitle={NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models},
year={2024},
url={https://openreview.net/forum?id=wkQy8mLIb9}
}
We lack a systematic understanding of the effects of fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback), particularly on tasks outside the narrow fine-tuning distribution. In a simplified scenario, we demonstrate that improving performance on fine-tuning tasks comes at the expense of other pretraining capabilities. We hypothesize that models implicitly infer the task of the prompt and that fine-tuning skews this inference towards fine-tuning tasks. We find that artificially making the task look farther from the fine-tuning distribution while requiring the same capability can recover some of the pretraining capabilities on our synthetic setup. Since real fine-tuning distributions are predominantly English, we apply conjugate prompting to recover pretrained capabilities in LLMs by simply translating the prompts to different languages. This allows us to recover the in-context learning abilities lost via instruction tuning, and more concerningly, recover harmful content generation suppressed by safety fine-tuning in chatbots like ChatGPT.
Understanding Catastrophic Forgetting in Language Models via Implicit Inference
Suhas Kotha Jacob Springer Aditi Raghunathan