@inproceedings{zheng-etal-2024-decompose,
    title = "Decompose, Prioritize, and Eliminate: Dynamically Integrating Diverse Representations for Multimodal Named Entity Recognition",
    author = "Zheng, Zihao  and
      Zhang, Zihan  and
      Wang, Zexin  and
      Fu, Ruiji  and
      Liu, Ming  and
      Wang, Zhongyuan  and
      Qin, Bing",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.403",
    pages = "4498--4508",
    abstract = "Multi-modal Named Entity Recognition, a fundamental task for multi-modal knowledge graph construction, requires integrating multi-modal information to extract named entities from text. Previous research has explored the integration of multi-modal representations at different granularities. However, they struggle to integrate all these multi-modal representations to provide rich contextual information to improve multi-modal named entity recognition. In this paper, we propose DPE-MNER, which is an iterative reasoning framework that dynamically incorporates all the diverse multi-modal representations following the strategy of {``}decompose, prioritize, and eliminate{''}. Within the framework, the fusion of diverse multi-modal representations is \textbf{decomposed} into hierarchically connected fusion layers that are easier to handle. The incorporation of multi-modal information \textbf{prioritizes} transitioning from {``}easy-to-hard{''} and {``}coarse-to-fine{''}. The explicit modeling of cross-modal relevance \textbf{eliminate} the irrelevances that will mislead the MNER prediction. Extensive experiments on two public datasets have demonstrated the effectiveness of our approach.",
}

Multi-modal Named Entity Recognition, a fundamental task for multi-modal knowledge graph construction, requires integrating multi-modal information to extract named entities from text. Previous research has explored the integration of multi-modal representations at different granularities. However, they struggle to integrate all these multi-modal representations to provide rich contextual information to improve multi-modal named entity recognition. In this paper, we propose DPE-MNER, which is an iterative reasoning framework that dynamically incorporates all the diverse multi-modal representations following the strategy of {``}decompose, prioritize, and eliminate{''}. Within the framework, the fusion of diverse multi-modal representations is \textbf{decomposed} into hierarchically connected fusion layers that are easier to handle. The incorporation of multi-modal information \textbf{prioritizes} transitioning from {``}easy-to-hard{''} and {``}coarse-to-fine{''}. The explicit modeling of cross-modal relevance \textbf{eliminate} the irrelevances that will mislead the MNER prediction. Extensive experiments on two public datasets have demonstrated the effectiveness of our approach.
Zheng, Zihao Zhang, Zihan Wang, Zexin Fu, Ruiji Liu, Ming Wang, Zhongyuan Qin, Bing
Decompose, Prioritize, and Eliminate: Dynamically Integrating Diverse Representations for Multimodal Named Entity Recognition