We extracted hidden state representations from each layer of four self-supervised models: APC [1]}, [2]}, wav2vec [3]}, wav2vec 2.0 [4]}, and HuBERT [5]}.
These models are pre-trained on 960 hours of speech from LibriSpeech [6]}, a corpus of English audiobooks from the LibriVox project.
The SSL models are not finetuned on any annotated samples.
See table:model-config for an overview of model architectures and training methods. In the main analyses, we evaluate the output of CNN encoders and all layers of the contextualizer.
