For the large high-capacity wav2vec 2.0 models, we note low accuracies without finetuning.
Other works corroborate this finding and ascribe it to the problem-agnostic pre-training, and have shown that certain audio features are more easily accessible from the middle layers in very deep transformer models than the output layer [1]}, [2]}. Finetuning (with graphemic transcripts) alleviates this discrepancy and gives an improved accuracy over the base models with the large models. It seems that the large model also benefits more from a large amount of pre-training data, as the LibriVox and XLSR model show.
