Across models, we see that the best layer from each SSL and Deep Speech 2 layer outperform the auditory baselines (articulation and spectrotemporal modulation). Despite being trained on the same amount of data, the best layers of SSL models wav2vec 2.0 and HuBERT outperform the fully supervised Deep Speech 2 model that was used in prior work investigating speech processing across cortex [1]}. Further, the best layer of HuBERT approaches the performance of the word embedding model, despite not receiving any explicit supervision on lexical or semantic information.
fig:encoding-perf-layersB shows average voxel prediction performance only for voxels within auditory cortex (AC), which is thought to be responsible for extracting higher-level features such as words from incoming acoustic information [2]}. Here, most SSL layers and supervised Deep Speech 2 layers outperform all of the hand-constructed feature spaces, including word embeddings. The best encoding model overall is the same, however: layer 9 of HuBERT.
<FIGURE>