@inproceedings{
shen2024neurovision,
title={Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction},
author={Guobin Shen and Dongcheng Zhao and Xiang He and Linghao Feng and Yiting Dong and Jihang Wang and Qian Zhang and Yi Zeng},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=ohi00YhT3T}
}
Decoding non-invasive brain recordings is pivotal for advancing our understanding of human cognition but faces challenges due to individual differences and complex neural signal representations. Traditional methods often require customized models and extensive trials, lacking interpretability in visual reconstruction tasks. Our framework integrates 3D brain structures with visual semantics using a *Vision Transformer 3D*. This unified feature extractor efficiently aligns fMRI features with multiple levels of visual embeddings, eliminating the need for subject-specific models and allowing extraction from single-trial data. The extractor consolidates multi-level visual features into one network, simplifying integration with Large Language Models (LLMs). Additionally, we have enhanced the fMRI dataset with diverse fMRI-image-related textual data to support multimodal large model development. Integrating with LLMs enhances decoding capabilities, enabling tasks such as brain captioning, complex reasoning, concept localization, and visual reconstruction. Our approach demonstrates superior performance across these tasks, precisely identifying language-based concepts within brain signals, enhancing interpretability, and providing deeper insights into neural processes. These advances significantly broaden the applicability of non-invasive brain decoding in neuroscience and human-computer interaction, setting the stage for advanced brain-computer interfaces and cognitive models.
Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction
Guobin Shen Dongcheng Zhao Xiang He Linghao Feng Yiting Dong Jihang Wang Qian Zhang Yi Zeng