Effectiveness of self supervised pre training for speech recognition
We compare self supervised representation learning algorithms which either explicitly quantize the audio data or learn representations without quantization. We find the former to be more accurate since it builds a good vocabulary of the data through vq wav2vec [1] to enable learning of effective representations in subsequent BERT training. Different to previous work, we directly fine tune the pre trained BERT models on transcribed speech using a Connectionist Temporal Classification (CTC) loss instead of feeding the representations into a task specific model. We also propose a BERT style model learning directly from the continuous audio data and compare pre training on raw audio to spectral features. Fine tuning a BERT model on 10 hour of labeled Librispeech data with a vq wav2vec vocabulary is almost as good as the best known reported system trained on 100 hours of labeled data on testclean, while achieving a 25% WER reduction on test other. When using only 10 minutes of labeled data, WER is 25.2 on test other and 16.3 on test clean. This demonstrates that self supervision can enable speech recognition systems trained on a near zero amount of transcribed data.
ArXiv