@inproceedings{li-etal-2024-assessing,
    title = "Assessing Image-Captioning Models: A Novel Framework Integrating Statistical Analysis and Metric Patterns",
    author = "Li, Qiaomu  and
      Xie, Ying  and
      Grundlingh, Nina  and
      Chawan, Varsha Rani  and
      Wang, Cody",
    editor = "Malmasi, Shervin  and
      Fetahu, Besnik  and
      Ueffing, Nicola  and
      Rokhlenko, Oleg  and
      Agichtein, Eugene  and
      Guy, Ido",
    booktitle = "Proceedings of the Seventh Workshop on e-Commerce and NLP @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.ecnlp-1.9",
    pages = "79--87",
    abstract = "In this study, we present a novel evaluation framework for image-captioning models that integrate statistical analysis with common evaluation metrics, utilizing two popular datasets, FashionGen and Amazon, with contrasting dataset variation to evaluate four models: Video-LLaVa, BLIP, CoCa and ViT-GPT2. Our approach not only reveals the comparative strengths of models, offering insights into their adaptability and applicability in real-world scenarios but also contributes to the field by providing a comprehensive evaluation method that considers both statistical significance and practical relevance to guide the selection of models for specific applications. Specifically, we propose Rank Score as a new evaluation metric that is designed for e-commerce image search applications and employ CLIP Score to quantify dataset variation to offer a holistic view of model performance.",
}

In this study, we present a novel evaluation framework for image-captioning models that integrate statistical analysis with common evaluation metrics, utilizing two popular datasets, FashionGen and Amazon, with contrasting dataset variation to evaluate four models: Video-LLaVa, BLIP, CoCa and ViT-GPT2. Our approach not only reveals the comparative strengths of models, offering insights into their adaptability and applicability in real-world scenarios but also contributes to the field by providing a comprehensive evaluation method that considers both statistical significance and practical relevance to guide the selection of models for specific applications. Specifically, we propose Rank Score as a new evaluation metric that is designed for e-commerce image search applications and employ CLIP Score to quantify dataset variation to offer a holistic view of model performance.
Li, Qiaomu Xie, Ying Grundlingh, Nina Chawan, Varsha Rani Wang, Cody
Assessing Image-Captioning Models: A Novel Framework Integrating Statistical Analysis and Metric Patterns