For the phoneme and word classification tasks, we developed a baseline that predicts the most frequent output category in the training set for all test set examples. All layers in the four SSL models beat the baseline. We do not visualize the word classification baseline perplexity since it is significantly worse than the highest layer PPX. For the word embedding MLP probe, we constructed two different baselines- randomly sampling embedding vectors from a normal distribution (“random”) and randomly shuffling the GloVe embeddings between words (“shuffle”). All layers in HuBERT beat the baseline with a considerable gap. Overall, the substantial differences in the performance of probing baselines and the SSL model layers suggest that the probes have high selectivity [1]}.
