Decoding Natural Sounds in Early “Visual” Cortex of Congenitally Blind Individuals

Summary Complex natural sounds, such as bird singing, people talking, or traffic noise, induce decodable fMRI activation patterns in early visual cortex of sighted blindfolded participants [1]. That is, early visual cortex receives non-visual and potentially predictive information from audition. However, it is unclear whether the transfer of auditory information to early visual areas is an epiphenomenon of visual imagery or, alternatively, whether it is driven by mechanisms independent from visual experience. Here, we show that we can decode natural sounds from activity patterns in early “visual” areas of congenitally blind individuals who lack visual imagery. Thus, visual imagery is not a prerequisite of auditory feedback to early visual cortex. Furthermore, the spatial pattern of sound decoding accuracy in early visual cortex was remarkably similar in blind and sighted individuals, with an increasing decoding accuracy gradient from foveal to peripheral regions. This suggests that the typical organization by eccentricity of early visual cortex develops for auditory feedback, even in the lifelong absence of vision. The same feedback to early visual cortex might support visual perception in the sighted [1] and drive the recruitment of this area for non-visual functions in blind individuals [2, 3].


In Brief
Natural sounds can be distinguished based on early visual cortex activity in sighted people. Is this effect driven by visual imagery? Vetter et al. report successful sound decoding, increasing from fovea to periphery, in people blind from birth, proving that visual imagery is not necessary for sound representation in these early visual areas.

SUMMARY
Complex natural sounds, such as bird singing, people talking, or traffic noise, induce decodable fMRI activation patterns in early visual cortex of sighted blindfolded participants [1]. That is, early visual cortex receives non-visual and potentially predictive information from audition. However, it is unclear whether the transfer of auditory information to early visual areas is an epiphenomenon of visual imagery or, alternatively, whether it is driven by mechanisms independent from visual experience. Here, we show that we can decode natural sounds from activity patterns in early ''visual'' areas of congenitally blind individuals who lack visual imagery. Thus, visual imagery is not a prerequisite of auditory feedback to early visual cortex. Furthermore, the spatial pattern of sound decoding accuracy in early visual cortex was remarkably similar in blind and sighted individuals, with an increasing decoding accuracy gradient from foveal to peripheral regions. This suggests that the typical organization by eccentricity of early visual cortex develops for auditory feedback, even in the lifelong absence of vision. The same feedback to early visual cortex might support visual perception in the sighted [1] and drive the recruitment of this area for non-visual functions in blind individuals [2,3].

Decoding of Natural Sounds in Congenitally Blind Individuals
To investigate the presence of sound representation in early visual cortex of blind individuals, we acquired fMRI data from 8 congenitally blind participants listening attentively to natural sounds. We used three natural sounds (one exemplar each): a forest scene (bird singing and a stream); a crowd scene (people talking without clear semantic information); and a street scene (traffic noise with cars and motorbikes; Figure 1A). We derived boundaries of early visual areas (V1, V2, and V3) and their foveal and peripheral regions, using cortex-based alignment and overlaying probabilistic retinotopic maps from sighted participants onto reconstructed brain surfaces of blind participants (Figure 1B). Using multi-variate pattern analysis (MVPA), we decoded the three different sounds from fMRI activity patterns in the corresponding early ''visual'' areas of blind individuals [1,4,5] (STAR Methods).
We found that we can decode natural sounds significantly above chance in all early visual areas (all early visual areas [EVCs] together, V1, V2, and V3) in the congenitally blind group ( Figure 1C; all p = 0.001 from permutation testing), with remarkable consistency across individual participants ( Figure 2). Confusion matrices illustrate similar classifier's predictions for each of the three sounds included in the experiment ( Figure S1). Unsurprisingly, sound decoding worked very well in auditory cortex (positive control) but at chance level in motor cortex (negative control; Figure 1C). The successful decoding of sounds in early visual areas mirrors the results previously found in a group of sighted participants ( Figure 1D; data taken from [1]). In order to investigate the role of visual imagery in the transfer of auditory information to early visual cortex, we previously conducted a series of control experiments in the sighted, concluding that visual imagery is unlikely to fully explain the observed pattern of activity in early visual areas [1]. However, we also concluded that the influence of visual imagery on auditory-induced activation patterns in visual cortex could not be ruled out entirely. Given that all blind participants lacked sight from birth on, and therefore lacked visual imagery, our current results in the blind group demonstrate that visual imagery is not a prerequisite of auditory feedback to early visual cortex.
Next, we looked at the eccentricity pattern of sound decoding accuracy in early visual cortex. We found that, in early visual cortex of blind individuals, sound decoding accuracy increased from foveal to peripheral regions, as indicated by a significant eccentricity effect in an eccentricity by area ANOVA ( Figure 1E; main effect of eccentricity: F(2,14) = 9.69, p = 0.002, partial eta squared = 0.58; all other ANOVA effects: p > 0.25) and a significant linear contrast for the eccentricity factor (F(1,7) = 14.72; p = 0.006; partial eta squared = 0.68). The observed gradient of decoding accuracy is comparable to the one previously found for sighted individuals ( Figure 1F; data taken from [1]). Thus, a spatial pattern of auditory feedback modulation of early visual cortex activity is present in the blind group and similar to the one found in the sighted group. This suggests that the typical organization of early visual areas by eccentricity develops even despite the life-long absence of vision.
A whole-brain searchlight analysis in the blind group ( Figure 3) revealed above-chance sound decoding in multisensory areas, such as superior temporal sulcus and middle temporal gyrus, similarly as shown previously in sighted participants [1]. This

. Experimental Design and Classification Results
(A) Congenitally blind individuals (n = 8) participated in an fMRI experiment in which they were listening to three natural sounds interleaved with silent periods (apart from MRI scanner noise). Multi-voxel pattern analysis (MVPA) was used to decode the sounds from participants' early visual cortex activity patterns. (B) In a separate fMRI session, retinotopic mapping was performed for a group of sighted participants to individually define early visual areas V1, V2, and V3. These retinotopically defined regions of interest (ROIs) were then mapped onto a cortical reconstruction of each blind participant using cortex-based alignment. Aligned ROIs were converted into maximum probability maps that were then used in the data analysis as early visual ROIs for blind participants. Additionally, auditory cortex and motor cortex ROIs were defined for each participant using brain atlases. (C and D) Mean classification accuracy of the classifier distinguishing the three natural sounds in the different ROIs in (C) blind participants and (D) sighted participants. The data for sighted participants (n = 10) were taken from [1] (the results for auditory cortex and motor cortex were recalculated within the same ROIs that were used for blind participants). Early visual cortex (EVC) contains V1, V2, and V3. AC, auditory cortex; MC, motor cortex. Chance level (one out of three) is marked with a black line. Error bars indicate SEM. Testing against chance level was performed with a permutation analysis. Results for V1, V2, and V3 were corrected for multiple comparisons, within each group, with a single threshold test. *p < 0.05; **p < 0.01; ***p = 0.001. (E and F) Mean classification accuracies for all visual ROIs divided into three eccentricities (fovea, periphery, and far periphery) in both (E) the blind and (F) sighted group. Results were corrected for multiple comparisons within each group using the false discovery rate. *p < 0.05; **p < 0.01. See also Figures S1 and S3 and Table S2.

OPEN ACCESS
suggests that sound-related information is represented in largely the same cortical network in both populations. When we assessed the significance of the searchlight results in a non-parametric group model, which allows the use of a more sensitive voxelwise statistical threshold than parametric models [6], we also detected above-chance decoding of sounds in ventral visual stream areas of the blind group, e.g., the lateral occipital complex (LOC), parahippocampal place area (PPA), and fusiform face area (FFA) ( Figure S2). The sounds used in this experiment convey different kinds of categorical information, e.g., they are animate or inanimate. Thus, this result is in line with previous findings of preserved category preference in the ventral visual stream areas in congenitally blind individuals [7][8][9][10][11].
On the univariate level ( Figure S3), none of the three sounds elicited substantial blood-oxygen-level-dependent (BOLD) response for the contrast sound > rest in early visual areas, and most importantly, no sound elicited a higher or lower univariate response than another (see region of interest [ROI] analysis presented in Figure S3C: Wilcoxon tests, all p > 0.1; similar effects in the sighted, see Supplemental Information of [1]). Therefore, all decoding effects from the MVPA were driven by small, subthreshold activity differences across voxels in each ROI.

Sound Decoding in the Blind Group and the Sighted Group-Direct Comparisons and Control Analyses
Overall, our results indicate a high level of correspondence between cortical organization for sound processing in congenitally blind (the current study) and sighted individuals [1]. This correspondence was observed despite the dramatic between-group difference in sensory experience and the fact that both groups were scanned using different MRI scanners (STAR Methods). However, the qualitative comparison of the results for both groups also hints at certain between-group differences, for example, higher decoding accuracy for early visual cortex of the blind group, relative to decoding accuracy obtained for the same region in the sighted group ( Figures 1C and 1D). Given that MVPA is typically performed on data that are normalized separately for each participant (see STAR Methods for details on the normalization procedure), which makes MVPA relatively invariant to differences in raw MRI signal that might arise from using different MRI scanners, we decided to supplement the main within-subject findings, reported above, with direct between-group comparisons.
In line with the pattern visible in qualitative comparisons, overall sound decoding accuracy in early visual cortex was significantly higher in the blind group than in the sighted group (62% versus 42%; Mann-Whitney U = 7; p = 0.002). A group 3 area 3 eccentricity ANOVA revealed a main effect of group (F(1,16) = 10.00; p = 0.006; partial eta squared = 0.39) and a main effect of eccentricity (F(2,32) = 8.62; p = 0.001; partial eta squared = 0.35), as well as a significant linear contrast for the eccentricity factor (F(1,16) = 14.22; p = 0.002; partial eta squared = 0.47). However, no group 3 eccentricity interaction was detected (F(2,32) = 1.89, p = 0.168, partial eta squared = 0.11; all other ANOVA effects: p > 0.08). Interestingly, in auditory cortex, we observed a trend toward lower decoding accuracy in the blind than in the sighted group (Mann-Whitney U = 20; p = 0.083), hinting at the possibility of the shift in distribution of sound representation, across auditory and visual cortices, in the blind group. To directly examine this possibility, we tested for the interaction between the group (blind participants and sighted participants) and the sensory region (early visual cortex and auditory cortex) in sound decoding accuracy. Because the data obtained for auditory cortex were not suitable for parametric ANOVA, due to ceiling effects ( Figures 1C and 1D), we performed a non-parametric interaction test: first, for each participant, we calculated the difference between decoding accuracy for auditory cortex and early visual cortex; subsequently, these difference scores were compared between groups using the Mann-Whitney test. The comparison was highly significant (U = 2.5; p < 0.001). This result indicates that visual deprivation might result in a sound representation that is more distributed across these two sensory regions [2,12,13]. As expected, we did not detect a between-group difference in decoding accuracy  Report for the motor cortex (Mann-Whitney U = 38.5; p > 0.25). These effects were confirmed in the whole-brain between-group comparison of searchlight results (Figure 4), in which significant results were detected only in visual cortex (higher decoding accuracy in the blind group) and in auditory cortex (higher decoding accuracy in the sighted group); no significant group effects were detected outside these two sensory regions, neither in the motor cortex nor in multisensory regions.
The spatial specificity of the observed between-group effects, as well as their opposite direction in visual and auditory cortices, suggest that these effects were not driven by different MRI scanners, as this would have most likely manifested itself in a more general way. Nevertheless, as a control analysis, we empirically investigated the characteristics of the BOLD signal in the early visual ROI that showed significantly better decoding in the blind group than in the sighted group in the searchlight analysis (Figure S4). As expected, the raw BOLD signal values obtained for both groups were clearly different, which is likely to be an effect of different MRI scanners ( Figure S4A). However, no betweengroup differences were observed in the values of normalized, Z scored contrast estimates (sound > rest) that served as input into the MVPA analyses, for neither of the three sounds nor their mean ( Figure S4B; Mann-Whitney tests, all p > 0.25). These results suggest that the normalization procedure applied to the data before MVPA decoding was successful and that the observed group differences in decoding accuracy were driven by sounds eliciting spatial activity patterns being distinguishable better or worse in one group than another rather than by global differences in BOLD signal. To illustrate this point further, we included individual means of the univariate responses to sounds in the early visual ROI as a covariate in the whole-brain betweengroup comparison of decoding accuracies obtained in the searchlight analysis. The between-group difference in visual and auditory cortices replicated, even with this covariate (Figure S4C). Thus, even when we regressed out potential residual differences in input values to the MVPA analysis that might have arisen due to different scanners (or other methodological issues), we still found robust between-group differences for early visual cortex and for auditory cortex. In summary, our control analyses suggest that the observed between-group effects arise from the subtle differences in activation patterns and not from a global difference in signal due to different MRI scanners.
Note that all within-subject effects, especially the successful sound decoding in early visual cortex and the fovea-periphery eccentricity effects, are independent of direct group comparisons and were confirmed in each group independently. One of the potential limitations of the within-subject comparisons is that the blind group is relatively small due to challenges related to finding congenitally blind participants with no or minimal light perception (STAR Methods). Nevertheless, individual data show that the within-subject group effects are remarkably consistent across blind participants ( Figure 2). Indeed, the main effects reported in our study-that is, the successful sound decoding in early visual cortex of the blind individuals, driven primarily by the decoding in the peripheries of this region-are clearly visible even at the level of individual p values, computed separately for each subject and ROI (Table S1).
To further confirm that the reported effects were not driven by outliers, we compared the decoding accuracies obtained for early visual areas of blind individuals to chance level using a bootstrapping procedure (STAR Methods). Because in this procedure various subsets of participants are included in a sample considered in each iteration (i.e., during each iteration, participants are drawn to a sample randomly, with replacements), results are unlikely to be affected by outliers. In this analysis, we largely replicated the results presented in Figure 1 (Table S2). Furthermore, to confirm that the fovea-periphery gradient effect is robust, we entered decoding accuracies for each of the eccentricities in the blind group (fovea, peripheries, and far peripheries; V1, V2, and V3 were combined for each eccentricity) into a nonparametric counterpart of a one-way repeated-measure ANOVA (the Friedman test). In line with the results of the parametric analysis, we observed a significant effect of eccentricity (chi-square = 9.48; p = 0.009). Overall, in combination with individual data, these control analyses show that our results in the blind group are robust and not driven by outliers.

DISCUSSION
Our study shows that natural sounds can be decoded successfully in early visual areas of congenitally blind individuals. Therefore, neither visual experience nor visual imagery is necessary for a transfer of sound-related information to early visual cortex. In fact, were visual imagery critical for this auditory feedback, it Brain regions in which a searchlight (cube with 7 voxels length-343 voxels in total) achieved abovechance classification accuracy of sounds in the blind group. Threshold: p < 0.001 voxelwise, corrected for multiple comparisons using cluster extent. The searchlight analysis was performed in the volume space, and results are displayed on a standard flattened cortical surface reconstruction for visualization purposes. White outlines represent combined Brodmann areas 17 and 18, as defined by a BrainVoyager brain atlas. CoS, collateral sulcus; EVC, early visual cortex; ITG, inferior temporal gyrus; ITS, inferior temporal sulcus; MTG, middle temporal gyrus; pSTS, posterior superior temporal sulcus; STS, superior temporal sulcus. See also Figure S2. Report should have boosted decoding accuracy in the sighted and lack of visual imagery should have diminished decoding accuracy in the blind group. Our results show that this was not the case, suggesting that, even in the sighted, sound decoding is not purely driven by visual imagery.
We also found that auditory feedback follows the organization of early visual cortex along an eccentricity gradient in the blind group, similarly as previously documented in the sighted [1]. This result suggests that the typical organization by eccentricity of early visual cortex can, to a large extent, develop even without any visual experience. Our finding complements previous results demonstrating retinotopic-like organization of functional connectivity of early visual cortex in blind individuals, highly similar to the ones observed in the sighted [14]. Higher sound decoding accuracy in peripheral areas of early visual cortex in both populations is also in line with previous evidence that, relative to the foveal part, the periphery has richer connections with numerous non-visual regions, particularly with higher level auditory areas and multisensory regions in the temporal and the parietal lobe [15][16][17][18]. Our results indicate that at least some of these connections might be preserved and functional, even in the lifelong absence of vision.
Sounds did not elicit differential univariate BOLD responses in early visual cortex, neither in the blind nor in the sighted population. This precludes the possibility that sound decoding worked on the basis of sounds attracting differential levels of attention or arousal, as attention strongly modulates univariate BOLD responses in early visual cortex [19,20]. All sounds were normalized for amplitude, and in early visual cortex of sighted individuals, sound decoding was successful even when the classifier was trained and tested on activity patterns elicited by sounds that differ in basic auditory features, as long as categorical distinctions between sounds were the same in the training and the testing set (e.g., ''animate'' versus ''inanimate''; see [1]). This generalization would not have worked if individual sound exemplars were distinguished only based on low-level acoustic (or visual) features. Instead, this result suggests that information reaching early visual cortex of sighted participants is, to an extent, content specific and possibly semantic. Given the high degree of correspondence between the results from sighted and blind participants, it is conceivable that the successful decoding in blind individuals may rely on the same type of high-level inputs.
In the sighted, auditory feedback to early visual cortex might be relevant for visual perception, in line with predictive coding theories [21]. In blind individuals, the same inputs might drive the recruitment of early visual areas for higher cognitive functions, such as verbal memory, language processing, or numerical computations [2,3,13]. A testable hypothesis is that this kind of recruitment could be most pronounced in peripheral parts of early visual cortex of blind individuals, as peripheral areas are primary receivers of auditory and, potentially, high-level information [15,16]. The foveal part of early visual cortex of blind individuals may, in turn, retain its typical preference for high-resolution spatial processing, such as recognizing Braille characters or localizing sound in space-tasks that are known to involve early visual cortex in both sighted and blind individuals [2,[22][23][24][25][26][27]. In summary, our results add to the growing body of evidence that early visual cortex might support other functions than purely feedforward processing, in the absence and presence of visual input.

STAR+METHODS
Detailed methods are provided in the online version of this paper and include the following:  Regions in which classification accuracy achieved by the searchlight (cube with 7 voxels length-343 voxels in total) was different in the blind and the sighted. Warm-color hues indicate higher decoding accuracy in blind participants, whereas coldcolor cues represent higher decoding accuracy in sighted participants. Data for sighted participants were taken from [1]. Threshold: p < 0.001 voxelwise, corrected for multiple comparisons using cluster extent. The searchlight analysis was performed in the volume space, and results are displayed on a standard flattened cortical surface reconstruction for visualization purposes. White outlines represent combined Brodmann areas 17 and 18, as defined by a BrainVoyager brain atlas. CoS, collateral sulcus; EVC, early visual cortex. See also Figure S4.

Lead Contact
For further information and requests for resources, Lead Contact will be Petra Vetter (petra.vetter@rhul.ac.uk; petra.vetter@unifr.ch).

Materials Availability
This study did not generate new materials.

EXPERIMENTAL MODEL AND SUBJECT DETAILS
Eight congenitally blind individuals with intact hearing (5 females, mean age 33.4 years, range 23-39 years, 4 left handers, mean education duration 13.6 years, range 12-17 years) participated in the study. Reasons for blindness were: microphthalmia in three participants of which one also had retinal detachment, retinopathy of prematurity in four participants, and enophthalmos in one participant. One blind participant had very faint light perception, all others had no light perception at all. All participants received detailed information of the study, signed informed consent and were paid for their participation. The study was approved by the Tel-Aviv Sourasky Medical Centre Ethics Committee, Israel.
Data from healthy participants with intact vision and hearing (n = 10, 7 females, mean age 24.1 years, range 20-33 years) were used as controls and taken from Experiment 1 of Vetter et al. [1], see Supplemental Online Information for details [1].

METHOD DETAILS
Stimuli and experimental procedures Stimuli and experimental procedures of the study with blind individuals were the same as in [1], Experiment 1 unless reported in the following. In brief, participants listened to one exemplar each of three natural sounds, traffic noise (a busy road with cars and motorbikes), a forest scene (birds singing and a stream) and a crowd scene (people talking in a foreign language without clear semantic