Notes Toward a Theory of Sensorimotor Understanding
Dan Lloyd is the Thomas C. Brownell Professor of Philosophy and a Professor of Neuroscience at Trinity College, Connecticut. He is the author/editor of 'Subjective Time: The philosophy, psychology, and neuroscience of temporality' (co-edited with Valtteri Arstila). In this article he discusses his developing research into the animation and sonification of brain activity.
Dan Lloyd
One of the dominant visual icons of our time is the brain. It's a challenge to depict. In itself, in reality, it's pinkish-gray and squishy, and there's nothing about its folds and ridges that reveals anything about how it works – how we work. So the imagery needs a boost, some pictorial add-in to convey that this particular organ is powerful and complicated. The resulting vivid images glow with the sheen of Science with a capital S, of Fact, of Knowledge, with all the implications of the esoteric: knowledge/power in the hands of an elite, with wide but unknown implications. These visual additions are symbolic and metaphorical. At their best they are works of art. Yet they are pictorially false. Their artistry doesn't illustrate actual brain function. They are rich in meaning as art, but scientifically silent.
With these iconic images in mind, consider Figure 1.
Figure 1. An independent component of brain function, isolated by some sophisticated statistics
Figure 1 is derived from a functional Magnetic Resonance Imaging (fMRI) session of one subject, awake but resting in a scanner. Full brain 3-d images were collected every .75 seconds; at each voxel (the 3-d equivalent of a pixel), the scanner detected the relative concentration of oxygenated hemoglobin, compared to deoxygenated hemoglobin; it is theorized (with pretty good confidence) that the concentration of oxygenated blood corresponds to the underlying metabolic expenditure of the million or so neurons in the neighborhood. (Like a muscle, neural tissue needs more oxygen when it works harder.) The voxels are tiny boxcars a few millimeters on a side, so each of these full brain images comprises several hundred thousand data points. The signal intensity at each of these voxels shifts by less than one percent with each passing second. It shifts at every voxel; to watch this in real time would be just barely more informative than the squishy gray itself.
Figure 1, then, involves several more steps of analysis. In this case, researchers applied a sophisticated statistical technique known as Independent Component Analysis (ICA). To understand ICA, an elaborate analogy: Suppose you are in a room full of highly talented madrigal singers. There are one hundred of them, each singing her or his line of a fifteen-part choral work. But they are not sitting together by section, but rather scattered about, some sections together but others interspersed. You want to know where the sopranos are. To make it more difficult, you don't know the song — so you can't walk around with the score and match each singer to written melody for her section. So, where are the sopranos? ICA will give you the answer. It's a form of "blind source separation," which will find the singers that are closest to singing in unison, and locate them on a map. The spatial region occupied by the sopranos is an Independent Component. ("Independent" means roughly that the melodic line sung by this section is maximally different from all the other section parts.) The ICA process allows you to divide the chorus into as many sections as you want. Possibly the first and second sopranos sing very similar parts. If we squeeze our singers into four groups, the various soprano sections will merge (after all, their parts are quite similar); or we can sort the room into fifty sections, in which case Amy and Anne, both with perfect pitch, wind up in their own section while Bev and Beth, who sing in unison but drift sharp and flat, might get a section of their own.
Returning to the voxels, we face a chorus of ninety thousand singers. We decide — arbitrarily — to sort them using ICA into fifteen sections. Figure 1 shows one of those sections. This is entirely data driven and thus has a better claim to truth than the fanciful images we know from popular media. It also has the look of scientific accuracy, with the intensity of signal mapped onto a continuum of red-orange-yellow, projected onto an anatomically accurate rendering of a brain in cross section. But this appearance is deceptive still, in several respects. First, it is a combination of two images, the blobby orange and the sharper gray. In fact, the brain activity detected by fMRI is quite blurry, the result of capturing images in the briefest time possible for this technology. But the gray outline is not a functional scan at all, but rather an anatomical ("structural") scan collected over several seconds — this anatomical image is much more precise than the functional image. Dropping the blob on top of the sharp edges gives the illusory impression that the blob edges are as precise as those in the gray background.
Since this mismatch of detail mainly conveys an impression of precision, it is perhaps not so consequential. The remaining distortions are more important. All are omissions, and plain within the framework of our choral analogy. We want to understand the elaborate music we hear, yet what ICA gives us is a static map. The dynamism of time has disappeared. Moreover, the blob of Figure 1 is now just one of fifteen blobs. Fourteen sections of the chorus have been shuffled into the wings. Finally, the figure shows just a slice of the brain. Choristers sitting above or below that slice have been disappeared.
These exclusions are obviously motivated by the need to convey the information in the image clearly and accurately. But now there is a further goal. Images like Figure 1 are meant to convey an understanding of some aspect of brain function. They are explanatory. In this respect, scientific images like Figure 1 and symbolic images like those that flood the popular press are of completely different types. Yet, as we've seen, Figure 1 omits as much (or more) than it includes. How then can it be explanatory?
Images like Figure 1 are the stock in trade of thousands of journal articles, and they are rarely offered as the sole documentation of an observation. The information in Figure 1 also appears verbally:
Figure 2. An observation in so many words
and in tabular form. (Figure 3)
Figure 3. Another way to display a blob
So, suppose a musicologist, studying the madrigals of an obscure Renaissance composer ("Smythe") reports: "The music of Smythe is strongly associated with loud sound from 1st through the 4th chairs of the third row of the choir." This report would do little to further the understanding of the performance or the music. The fMRI observations are similarly limited, conveying accurate and quite precise information but saying little about how it all works. Neither the brain images nor our imaginary musicology offer explanations. All of these types of representation (pictorial, verbal, tabular) step toward explanation when they are fit into a larger context. They convey understanding only in the context of an interpretation that restores their place in a complex whole.
Thus, a map of one row of the chorus, locating just the first sopranos, can help us understand the dynamics of the music only by adding back the dynamic elements that are missing. Or in other words, we embed the descriptions inside a story. In music, that story is the melody, counterpoint, and whatever else is encoded in a musical score. In cognitive neuroscience it's something more abstract. Here is one example, chosen more or less at random:
The auditory cortex represents spatial locations differently from other sensory modalities. While visual and tactile cortices utilize topographical space maps, for audition no such cortical map has been found. Instead, auditory cortical neurons have wide spatial receptive fields and together they form a population rate code of sound source location. Recent studies have shown that this code is modulated by task conditions so that during auditory tasks it provides better selectivity to sound source location than during idle listening. The goal of this study was to establish whether the neural representation of auditory space can also be influenced by task conditions involving other sensory modalities than hearing. Therefore,…
("Visual task enhances spatial selectivity in the human auditory cortex," Front. Neurosci., 27 March 2013 | https://doi.org/10.3389/fnins.2013.00044)
Now the blobs are interpreted as snapshots of a larger process. In the first wave of fMRI research, from around 1990 to 2010, the blobs were associated with particular behaviors or psychological states. Seeing a face would be linked to activity in one area of the brain; moving a finger activated its special area, and so forth. The grand picture was one of functional localization. This research is still very much in play, but even in the early years, researchers recognized that function wasn't fully localizable, but rather our mental lives and our behaviors were the result of the simultaneous interaction of many parts of the brain. For a more complete understanding of the brain, functional localization needed the addition of network dynamics. Blobs remain important, but now it's their interconnections that do the work. Contemporary research examines two aspects of the connections: functional connections and anatomical connections. Two blobs are functionally connected if they tend to glow in unison, like for example the first and second sopranos — still distinct from each other but much more distinct from the tenors and basses. Their functional connection derives from their synchronization: They synchronize, we assume, because of some sort of physical link connecting them. These physical links implement the functional connections via axons that crisscross the brain by the billions. (This is the white matter inside the gray matter of the cortex.) Projects now seek to identify the real physical circuitry of the brain underlying the functional connections. This circuit diagram, in all its intricacy, is known as the connectome. Figure 4 is a typical image of the connectome.
Figure 4. The Connectome
Its order is elusive, its look surreal. And, still — it's inert.
Typical brain images are inert for many reasons. Some are technical, but one is obvious: The medium, the printed page, is static. Not too long ago published images were in black and white. The widespread addition of color has ramped up the information an image can display. But the dynamics, the story, remain abstract, conceptual, and conveyed by words or numbers. These scientific illustrations have largely rejected a deliberate attempt to create works of art, choosing to favor the data over the visual metaphor. Nonetheless, as we have seen, they move us only marginally closer to scientific understanding.
Digital media can change that:
Sources and Resources: Notes toward a theory of sensorimotor understanding
The data that have been transformed into the video were provided by the Human Connectome Project (HCP), WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University. (For an introduction, see David Van Essen, et al, "The WU-Minn Human Connectome Project: An Overview," https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3724347/ ). The HCP ICA analysis is discussed in https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FIX . Figures 1 and 3 are screen shots from the Group ICA of fMRI Toolbox (GIFT, http://mialab.mrn.org/software/gift/ ). Figure 4 is reprinted from Stephan Gerhard et al, "The Connectome Viewer Toolkit: An open source framework to manage, analyze, and visualize connectomes," published in the open source journal, Frontiers in Neuroinformatics (2011) (https://www.frontiersin.org/files/Articles/10144/fninf-05-00003-HTML/image_m/fninf-05-00003-g012.jpg).
The visualization and sonification programs were written in Matlab by Dan Lloyd. The actual sounds were synthesized with Sweet Midi Player software (Roni Music, 2017, www.ronimusic.com).
Brain, Mind, Music, Neuroscience, Science, Technology, video
About Dan Lloyd
Dan Lloyd is the Thomas C. Brownell Professor of Philosophy and a Professor of Neuroscience at Trinity College, Connecticut. He is the author/editor of 'Subjective Time: The philosophy, psychology, and neuroscience of temporality' (co-edited with Valtteri Arstila), MIT Press, 2014; 'Radiant Cool: a novel theory of consciousness', an exploration of consciousness presented as noir detective fiction, MIT Press, 2004; and 'Simple Minds', a philosophical examination of scientific approaches to the mind and brain. Bradford Books/MIT Press, 1989. He has presented his animations and sonifications of brain activity in the US, Great Britain, and Europe.	View all posts with Dan Lloyd →
← Neurocosmologies
Making art with neuroscience →
The Lines that Connect
Issue | December 2018
Amanpreet Badhwar
Shadows in a Labyrinth
Stéphanie Morissette
Design and Neuroscience
Alexa Piotte
Neural Connections
Tyler Sloan
Neuroscience-Art
Julia Buntaine Hoel
Aesthetics and Memory
Neurocosmologies
Shanthi Chandrasekar
Cellular Kinesics
Tia Besser-Paul
Making art with neuroscience
Shima Rastegarnia
Guiding Memory
Rosi Maria Di Meglio
Josefina Maranzano
Neural Communications
Richard Bright