@inproceedings{
ma2024cognitionsupervised,
title={Cognition-Supervised Saliency Detection: Contrasting {EEG} Signals and Visual Stimuli},
author={Jun Ma and Tuukka Ruotsalo},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=0JO83loPPV}
}
Understanding human assessment of semantically salient parts of multimedia content is crucial for developing human-centric applications, such as annotation tools, search and recommender systems, and systems able to generate new media matching human interests. However, the challenge of acquiring suitable supervision signals to detect semantic saliency without extensive manual annotation remains significant. Here, we explore a novel method that utilizes signals measured directly from human cognition via electroencephalogram (EEG) in response to natural visual perception. These signals are used for supervising representation learning to capture semantic saliency. Through a contrastive learning framework, our method aligns EEG data with visual stimuli, capturing human cognitive responses without the need for any manual annotation. Our approach demonstrates that the learned representations closely align with human-centric notions of visual saliency and achieve competitive performance in several downstream tasks, such as image classification and generation. As a contribution, we introduce an open EEG/image dataset from 30 participants, to facilitate further research in utilizing cognitive signals for multimodal data analysis, studying perception, and developing models for cross-modal representation learning.
Cognition-Supervised Saliency Detection: Contrasting EEG Signals and Visual Stimuli
Jun Ma Tuukka Ruotsalo