@inproceedings{
wang2024eviledit,
title={EvilEdit: Backdooring Text-to-Image Diffusion Models in One Second},
author={Hao Wang and Shangwei Guo and Jialing He and Kangjie Chen and Shudong Zhang and Tianwei Zhang and Tao Xiang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=ibEaSS6bQn}
}
Text-to-image (T2I) diffusion models enjoy great popularity and many individuals and companies build their applications based on publicly released T2I diffusion models. Previous studies have demonstrated that backdoor attacks can elicit T2I diffusion models to generate unsafe target images through textual triggers. However, existing backdoor attacks typically demand substantial tuning data for poisoning, limiting their practicality and potentially degrading the overall performance of T2I diffusion models. To address these issues, we propose EvilEdit, a **training-free** and **data-free** backdoor attack against T2I diffusion models. EvilEdit directly edits the projection matrices in the cross-attention layers to achieve projection alignment between a trigger and the corresponding backdoor target. We preserve the functionality of the backdoored model using a protected whitelist to ensure the semantic of non-trigger words is not accidentally altered by the backdoor. We also propose a visual target attack EvilEdit$_{VTA}$, enabling adversaries to use specific images as backdoor targets. We conduct empirical experiments on Stable Diffusion and the results demonstrate that the EvilEdit can backdoor T2I diffusion models within **one second** with up to 100% success rate. Furthermore, our EvilEdit modifies only 2.2% of the parameters and maintains the model’s performance on benign prompts. Our code is available at [https://github.com/haowang-cqu/EvilEdit](https://github.com/haowang-cqu/EvilEdit).
EvilEdit: Backdooring Text-to-Image Diffusion Models in One Second
Hao Wang Shangwei Guo Jialing He Kangjie Chen Shudong Zhang Tianwei Zhang Tao Xiang