@inproceedings{
gong2024neuroclips,
title={NeuroClips: Towards High-fidelity and Smooth f{MRI}-to-Video Reconstruction},
author={Zixuan Gong and Guangyin Bao and Qi Zhang and Zhongwei Wan and Duoqian Miao and Shoujin Wang and Lei Zhu and Changwei Wang and Rongtao Xu and Liang Hu and Ke Liu and Yu Zhang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=8qu52Fl1Dt}
}
Reconstruction of static visual stimuli from non-invasion brain activity fMRI achieves great success, owning to advanced deep learning models such as CLIP and Stable Diffusion. However, the research on fMRI-to-video reconstruction remains limited since decoding the spatiotemporal perception of continuous visual experiences is formidably challenging. We contend that the key to addressing these challenges lies in accurately decoding both high-level semantics and low-level perception flows, as perceived by the brain in response to video stimuli. To the end, we propose NeuroClips, an innovative framework to decode high-fidelity and smooth video from fMRI. NeuroClips utilizes a semantics reconstructor to reconstruct video keyframes, guiding semantic accuracy and consistency, and employs a perception reconstructor to capture low-level perceptual details, ensuring video smoothness. During inference, it adopts a pre-trained T2V diffusion model injected with both keyframes and low-level perception flows for video reconstruction. Evaluated on a publicly available fMRI-video dataset, NeuroClips achieves smooth high-fidelity video reconstruction of up to 6s at 8FPS, gaining significant improvements over state-of-the-art models in various metrics, e.g., a 128% improvement in SSIM and an 81% improvement in spatiotemporal metrics. Our project is available at https://github.com/gongzix/NeuroClips.
NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction
Zixuan Gong Guangyin Bao Qi Zhang Zhongwei Wan Duoqian Miao Shoujin Wang Lei Zhu Changwei Wang Rongtao Xu Liang Hu Ke Liu Yu Zhang