Reconstructing perceived images based on brain signals measured with functional magnetic resonance imaging (fMRI) is a significant and meaningful task in brain-driven computer vision. However, the inconsistent distribution and representation between fMRI signals and visual images cause the heterogeneity gap, which makes it challenging to learn a reliable mapping between them. Moreover, considering that fMRI signals are extremely high-dimensional and contain a lot of visually-irrelevant information, effectively reducing the noise and encoding powerful visual representations for image reconstruction is also an open problem. We show that it is possible to overcome these challenges by learning a visually-relevant latent representation from fMRI signals guided by the corresponding visual features, and recovering the perceived images via adversarial learning. The resulting framework is called Dual-Variational Autoencoder Generative Adversarial Network (D-VAE GAN). By using a novel 3-stage training strategy, it encodes both cognitive and visual features via a dual structure variational autoencoder (D-VAE) to adapt cognitive features to visual feature space, and then learns to reconstruct perceived images with generative adversarial network (GAN). Extensive experiments on three fMRI recording datasets show that D-VAE GAN achieves more accurate visual reconstruction compared with the state-of-the-art methods.
Even though research on brain decoding has attracted increasing attention, a relatively limited number of studies focus on perceived image reconstructions to date. Electroencephalography (EEG) @cite_27 and fMRI are two most widely used neural signals in these decoding tasks. EEG has a high temporal resolution but insufficient spatial resolution, and thus is difficult to locate the active regions in the brain. Contrastively, fMRI is capable of providing more abundant spatial information for higher-precision decoding.
@cite_27
2013150886
Abstract Functional magnetic resonance imaging (fMRI) can provide maps of brain activation with millimeter spatial resolution but is limited in its temporal resolution to the order of seconds. Here, we describe a technique that combines structural and functional MRI with magnetoencephalography (MEG) to obtain spatiotemporal maps of human brain activity with millisecond temporal resolution. This new technique was used to obtain dynamic statistical parametric maps of cortical activity during semantic processing of visually presented words. An initial wave of activity was found to spread rapidly from occipital visual cortex to temporal, pariet al, and frontal areas within 185 ms, with a high degree of temporal overlap between different areas. Repetition effects were observed in many of the same areas following this initial wave of activation, providing evidence for the involvement of feedback mechanisms in repetition priming.