One approach to studying sensory representations in the brain is through “encoding models” [1]}. These predictive models of brain activity learn a mapping from stimulus features to responses measured using a neuroimaging technique like fMRI. In this work, we use SSL models to extract acoustic features of natural speech and in turn, build voxel-wise speech encoding models using data from an fMRI experiment (fig:intro). Our experiment comprised human participants passively listening to English-language narrative stories while their whole-brain fMRI BOLD activity was being recorded. Prior work on LM-based language encoding models has found performance differences across LM layers [2]}, [3]}, and layer-wise analyses of SSL models have shown that they capture different types of acoustic information [4]}. Motivated by these findings, we built separate encoding models for each layer in four SSL models — APC, wav2vec, wav2vec 2.0 and HuBERT (table:model-config). Their performance was compared to low-level acoustic baselines like Mel spectrograms and spectrotemporal features, mid-level phoneme articulations, and high-level semantic features.
