@inproceedings{
zeng2024hicescore,
title={{HICES}core: A Hierarchical Metric for Image Captioning Evaluation},
author={Zequn Zeng and Jianqiao Sun and Hao Zhang and Tiansheng Wen and Yudi Su and Yan Xie and Zhengjue Wang and Bo Chen},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=vite2JNeiD}
}
Image captioning evaluation metrics can be divided into two categories: reference-based metrics and reference-free metrics. However, reference-based approaches may struggle to evaluate descriptive captions with abundant visual details produced by advanced multimodal large language models, due to their heavy reliance on limited human-annotated references. In contrast, previous reference-free metrics have been proven effective via CLIP cross-modality similarity. Nonetheless, CLIP-based metrics, constrained by their solution of global image-text compatibility, often have a deficiency in detecting local textual hallucinations and are insensitive to small visual objects. Besides, their single-scale designs are unable to provide an interpretable evaluation process such as pinpointing the position of caption mistakes and identifying visual regions that have not been described. To move forward, we propose a novel reference-free metric for image captioning evaluation, dubbed Hierarchical Image Captioning Evaluation Score (HICE-S). By detecting local visual regions and textual phrases, HICE-S builds an interpretable hierarchical scoring mechanism, breaking through the barriers of the single-scale structure of existing reference-free metrics. Comprehensive experiments indicate that our proposed metric achieves the SOTA performance on several benchmarks, outperforming existing reference-free metrics like CLIP-S and PAC-S, and reference-based metrics like METEOR and CIDEr. Moreover, several case studies reveal that the assessment process of HICE-S on detailed texts closely resembles interpretable human judgments. The code is available in the supplementary.
HICEScore: A Hierarchical Metric for Image Captioning Evaluation
Zequn Zeng Jianqiao Sun Hao Zhang Tiansheng Wen Yudi Su Yan Xie Zhengjue Wang Bo Chen