Decoding naturalistic experiences from human brain activity via distributed representations of words
ABSTRACT Natural visual scenes induce rich perceptual experiences that are highly diverse from scene to scene and from person to person. Here, we propose a new framework for decoding such experiences using a distributed representation of words. We used functional magnetic resonance imaging (fMRI) to measure brain activity evoked by natural movie scenes. Then, we constructed a high dimensional feature space of perceptual experiences using skip gram, a state of the art distributed word embedding model. We built a decoder that associates brain activity with perceptual experiences via the distributed word representation. The decoder successfully estimated perceptual contents consistent with the scene descriptions by multiple annotators. Our results illustrate three advantages of our decoding framework: (1) three types of perceptual contents could be decoded in the form of nouns (objects) verbs (actions) and adjectives (impressions) contained in 10,000 vocabulary words; (2) despite using such a large vocabulary, we could decode novel words that were absent in the datasets to train the decoder; and (3) the inter individual variability of the decoded contents co varied with that of the contents of scene descriptions. These findings suggest that our decoding framework can recover diverse aspects of perceptual experiences in naturalistic situations and could be useful in various scientific and practical applications. HIGHLIGHTSA new decoding method that uses a distributed representation of words.Decoding of movie induced perceptions from cortical activity in the form of words.Decoding could infer object, action, and impression perception separately.Inter individual variability in decoding correlated with variability in perception.Our method provides a useful tool for scientific and practical applications.
NeuroImage