Improving Low-Resource Speech Recognition with Pretrained Speech Models: Continued Pretraining vs. Semi-Supervised Training
Self-supervised Transformer based models, such as wav2vec 2.0 and HuBERT,
have produced significant improvements over existing approaches to automatic
speech recognition (ASR). This is evident in the performance of the wav2vec 2.0
based pretrained XLSR-53 model across many languages when fine-tuned with
available labeled data. However, the performance from finetuning these models
can be dependent on the amount of in-language or similar-to-in-language data
included in the pretraining dataset. In this paper we investigate continued
pretraining (CoPT) with unlabeled in-language audio data on the XLSR-53
pretrained model in several low-resource languages. CoPT is more
computationally efficient than semi-supervised training (SST), the standard
approach of utilizing unlabeled data in ASR, since it omits the need for
pseudo-labeling of the unlabeled data. We show CoPT results in word error rates
(WERs), equal to or slightly better than using SST. In addition, we show that
using the CoPT model for pseudo-labeling, and using these labels in SST,
results in further improvements in WER.