Fine-tuning is dismissed as not effective for model editing
due to its poor performance compared to more specialized methods.
However, fine-tuning is simple, agnostic to the architectural details of the model being edited,
and able to leverage ongoing advances in standard training methods (e.g., PEFT) ,
making it an appealing choice for a model editor.
In this work, we show that pure fine-tuning can be a viable approach to model editing.
We propose a slight modification of naive fine-tuning with two key ingredients.
First, we optimize the conditional likelihood rather than the full likelihood.
Second, we augment the data with random paraphrases and facts to encourage generalization and locality.
Our experiments on ZsRE andCounterFactshow that this simple modification
allows fine-tuning to often match or outperform specialized editors in the edit score.