@inproceedings{
ma2024iebench,
title={I2{EB}ench: A Comprehensive Benchmark for Instruction-based Image Editing},
author={Yiwei Ma and Jiayi Ji and Ke Ye and Weihuang Lin and zhibin wang and Yonghan Zheng and Qiang Zhou and Xiaoshuai Sun and Rongrong Ji},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=1dpmeH6IHa}
}
Significant progress has been made in the field of Instruction-based Image Editing (IIE). However, evaluating these models poses a significant challenge. A crucial requirement in this field is the establishment of a comprehensive evaluation benchmark for accurately assessing editing results and providing valuable insights for its further development. In response to this need, we propose I2EBench, a comprehensive benchmark designed to automatically evaluate the quality of edited images produced by IIE models from multiple dimensions. I2EBench consists of 2,000+ images for editing, along with 4,000+ corresponding original and diverse instructions. It offers three distinctive characteristics: 1) Comprehensive Evaluation Dimensions: I2EBench comprises 16 evaluation dimensions that cover both high-level and low-level aspects, providing a comprehensive assessment of each IIE model. 2) Human Perception Alignment: To ensure the alignment of our benchmark with human perception, we conducted an extensive user study for each evaluation dimension. 3) Valuable Research Insights: By analyzing the advantages and disadvantages of existing IIE models across the 16 dimensions, we offer valuable research insights to guide future development in the field. We will open-source I2EBench, including all instructions, input images, human annotations, edited images from all evaluated methods, and a simple script for evaluating the results from new IIE models. The code, dataset, and generated images from all IIE models are provided in GitHub: https://github.com/cocoshe/I2EBench.
I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing
Yiwei Ma Jiayi Ji Ke Ye Weihuang Lin zhibin wang Yonghan Zheng Qiang Zhou Xiaoshuai Sun Rongrong Ji