@inproceedings{
li2024generative,
title={Generative Multimodal Data Augmentation for Low-Resource Multimodal Named Entity Recognition},
author={Ziyan Li and Jianfei Yu and Jia Yang and Wenya Wang and Li Yang and Rui Xia},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=QEJxw4hiFn}
}
As an important task in multimodal information extraction, Multimodal Named Entity Recognition (MNER) has recently attracted considerable attention.
One key challenge of MNER lies in the lack of sufficient fine-grained annotated data, especially in low-resource scenarios.
Although data augmentation is a widely used technique to tackle the above issue, it is challenging to simultaneously generate synthetic text-image pairs and their corresponding high-quality entity annotations.
In this work, we propose a novel Generative Multimodal Data Augmentation (GMDA) framework for MNER, which contains two stages: Multimodal Text Generation and Multimodal Image Generation.
Specifically, we first transform each annotated sentence into a linearized labeled sequence, and then train a Label-aware Multimodal Large Language Model (LMLLM) to generate the labeled sequence based on a label-aware prompt and its associated image.
After using the trained LMLLM to generate synthetic labeled sentences, we further employ a Stable Diffusion model to generate the synthetic images that are semantically related to these sentences.
Experimental results on three benchmark datasets demonstrate the effectiveness of the proposed GMDA framework, which consistently boosts the performance of several competitive methods for two subtasks of MNER in both full-supervision and low-resource settings.
Generative Multimodal Data Augmentation for Low-Resource Multimodal Named Entity Recognition
Ziyan Li Jianfei Yu Jia Yang Wenya Wang Li Yang Rui Xia