fig:probing shows the normalized probing performance of different tasks as a function of SSL model depth.
These results suggest that the SSL layers recapitulate the putative stages of speech processing [1]}, [2]}, [3]}. For most models the simplest features (FBANK) are best captured by the lowest layers, slightly more complex features (spectrotemporal) by the lower-middle layers, and high-level features (words) by the upper-middle layers. Yet phonemes, which are generally thought to fall between spectrotemporal and word-level features in speech analysis [1]}, are best captured by the uppermost layers in wav2vec 2.0 and HuBERT. This may reflect how phoneme perception is influenced by word context [5]} rather than being purely based on acoustics. This effect appears to mirror human behavior despite the fact that these models have no explicit representation of words or phonemes.
