@inproceedings{
wu2024hypergraph,
title={Hypergraph Multi-modal Large Language Model: Exploiting {EEG} and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding},
author={Minghui Wu and Chenxu Zhao and Anyang Su and Donglin Di and Tianyu Fu and Da An and Min He and Ya Gao and Meng Ma and Kun Yan and Ping Wang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=mFy8n4Gdc9}
}
Understanding of video creativity and content often varies among individuals, with differences in focal points and cognitive levels across different ages, experiences, and genders. There is currently a lack of research in this area, and most existing benchmarks suffer from several drawbacks: 1) a limited number of modalities and answers with restrictive length;  2) the content and scenarios within the videos are excessively monotonous, transmitting allegories and emotions that are overly simplistic. To bridge the gap to real-world applications, we introduce a large-scale Video $\textbf{S}$ubjective $\textbf{M}$ulti-modal $\textbf{E}$valuation dataset, namely Video-SME.  Specifically, we collected real changes in Electroencephalographic (EEG) and eye-tracking regions from different demographics while they viewed identical video content. Utilizing this multi-modal dataset, we developed tasks and protocols to analyze and evaluate the extent of cognitive understanding of video content among different users.  Along with the dataset, we designed a $\textbf{H}$ypergraph $\textbf{M}$ulti-modal $\textbf{L}$arge $\textbf{L}$anguage $\textbf{M}$odel (HMLLM) to explore the associations among different demographics, video elements, EEG and eye-tracking indicators. HMLLM could bridge semantic gaps across rich modalities and integrate information beyond different modalities to perform logical reasoning. Extensive experimental evaluations on Video-SME and other additional video-based generative performance benchmarks demonstrate the effectiveness of our method. The code and dataset are available at https://github.com/mininglamp-MLLM/HMLLM
Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding
Minghui Wu Chenxu Zhao Anyang Su Donglin Di Tianyu Fu Da An Min He Ya Gao Meng Ma Kun Yan Ping Wang