@inproceedings{
he2024focus,
title={Focus \& Gating: A Multimodal Approach for Unveiling Relations in Noisy Social Media},
author={Liang He and Hongke Wang and Zhen Wu and Jianbing Zhang and Xinyu Dai and Jiajun Chen},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=qjUTgApgAN}
}
With the rise of multimedia-driven content on the internet, multimodal relation extraction has gained significant importance in various domains, such as intelligent search and multimodal knowledge graph construction. Social media, as a rich source of image-text data, plays a crucial role in populating knowledge bases. However, the noisy information present in social media data poses a challenge in multimodal relation extraction. Current methods focus on extracting relevant information from images to improve model performance but often overlook the importance of global image information. In this paper, we propose a novel multimodal relation extraction method, named FocalMRE, which leverages image focal augmentation, focal attention, and gating mechanisms. FocalMRE enables the model to concentrate on the image's focal regions while effectively utilizing the global information in the image. Through gating mechanisms, FocalMRE optimizes the multimodal fusion strategy, allowing the model to select the most relevant augmented regions for overcoming noise interference in relation extraction. The experimental results on the public MNRE dataset reveal that our proposed method exhibits robust and significant performance advantages in the multimodal relation extraction task, especially in scenarios with high noise, long-tail distributions, and limited resources.
Focus     Gating: A Multimodal Approach for Unveiling Relations in Noisy Social Media
Liang He Hongke Wang Zhen Wu Jianbing Zhang Xinyu Dai Jiajun Chen