Reconstructing perceived images based on brain signals measured with functional magnetic resonance imaging (fMRI) is a significant and meaningful task in brain-driven computer vision. However, the inconsistent distribution and representation between fMRI signals and visual images cause the heterogeneity gap, which makes it challenging to learn a reliable mapping between them. Moreover, considering that fMRI signals are extremely high-dimensional and contain a lot of visually-irrelevant information, effectively reducing the noise and encoding powerful visual representations for image reconstruction is also an open problem. We show that it is possible to overcome these challenges by learning a visually-relevant latent representation from fMRI signals guided by the corresponding visual features, and recovering the perceived images via adversarial learning. The resulting framework is called Dual-Variational Autoencoder Generative Adversarial Network (D-VAE GAN). By using a novel 3-stage training strategy, it encodes both cognitive and visual features via a dual structure variational autoencoder (D-VAE) to adapt cognitive features to visual feature space, and then learns to reconstruct perceived images with generative adversarial network (GAN). Extensive experiments on three fMRI recording datasets show that D-VAE GAN achieves more accurate visual reconstruction compared with the state-of-the-art methods.
Even though research on brain decoding has attracted increasing attention, a relatively limited number of studies focus on perceived image reconstructions to date. Electroencephalography (EEG) @cite_27 and fMRI are two most widely used neural signals in these decoding tasks. EEG has a high temporal resolution but insufficient spatial resolution, and thus is difficult to locate the active regions in the brain. Contrastively, fMRI is capable of providing more abundant spatial information for higher-precision decoding.
@cite_27
2013150886 2112532472 2097267381 2112180451
Abstract Functional magnetic resonance imaging (fMRI) can provide maps of brain activation with millimeter spatial resolution but is limited in its temporal resolution to the order of seconds. Here, we describe a technique that combines structural and functional MRI with magnetoencephalography (MEG) to obtain spatiotemporal maps of human brain activity with millisecond temporal resolution. This new technique was used to obtain dynamic statistical parametric maps of cortical activity during semantic processing of visually presented words. An initial wave of activity was found to spread rapidly from occipital visual cortex to temporal, pariet al, and frontal areas within 185 ms, with a high degree of temporal overlap between different areas. Repetition effects were observed in many of the same areas following this initial wave of activation, providing evidence for the involvement of feedback mechanisms in repetition priming. Over the past decade, functional Magnetic Resonance Imaging (fMRI) has emerged as a powerful new instrument to collect vast quantities of data about activity in the human brain. A typical fMRI experiment can produce a three-dimensional image related to the human subject's brain activity every half second, at a spatial resolution of a few millimeters. As in other modern empirical sciences, this new instrumentation has led to a flood of new data, and a corresponding need for new data analysis methods. We describe recent research applying machine learning methods to the problem of classifying the cognitive state of a human subject based on fRMI data observed over a single time interval. In particular, we present case studies in which we have successfully trained classifiers to distinguish cognitive states such as (1) whether the human subject is looking at a picture or a sentence, (2) whether the subject is reading an ambiguous or non-ambiguous sentence, and (3) whether the word the subject is viewing is a word describing food, people, buildings, etc. This learning problem provides an interesting case study of classifier learning from extremely high dimensional (105 features), extremely sparse (tens of training examples), noisy data. This paper summarizes the results obtained in these three case studies, as well as lessons learned about how to successfully apply machine learning methods to train classifiers in such settings. SUMMARY Perceptualexperienceconsistsofanenormousnumber of possible states. Previous fMRI studies have predicted a perceptual state by classifying brain activity into prespecified categories. Constraint-free visual image reconstruction is more challenging, as itisimpracticaltospecifybrainactivityforallpossible images.Inthisstudy,wereconstructedvisualimages by combining local image bases of multiple scales, whose contrasts were independently decoded from fMRI activity by automatically selecting relevant voxels and exploiting their correlated patterns. Binarycontrast, 10 3 10-patch images (2 100 possible states) were accurately reconstructed without any image prior on a single trial or volume basis by measuring brain activity only for several hundred random images. Reconstruction was also used to identify the presented image among millions of candidates. Theresultssuggestthatourapproachprovidesaneffective means to read out complex perceptual states from brain activity while discovering information representation in multivoxel patterns. Summary Recent studies have used fMRI signals from early visual areas to reconstruct simple geometric patterns. Here, we demonstrate a new Bayesian decoder that uses fMRI signals from early and anterior visual areas to reconstruct complex natural images. Our decoder combines three elements: a structural encoding model that characterizes responses in early visual areas, a semantic encoding model that characterizes responses in anterior visual areas, and prior information about the structure and semantic content of natural images. By combining all these elements, the decoder produces reconstructions that accurately reflect both the spatial structure and semantic category of the objects contained in the observed natural image. Our results show that prior information has a substantial effect on the quality of natural image reconstructions. We also demonstrate that much of the variance in the responses of anterior visual areas to complex natural images is explained by the semantic category of the image alone.