In this work, we compare several popular self-supervised pre-training methods when applied to Flemish. First of all, we look at the applicability of off-the-shelf models that are pre-trained on English and assess the transferability to Flemish. This would be convenient for several research domains and technological applications. Additionally it would eliminate the need for large computational resources necessary to pre-train these models, which scales with the model size (e.g. the high-capacity wav2vec 2.0 model [1]}) and for many models also with the amount of data. Second, we examine the importance of matching the pre-training language to the target language as opposed to the amount of data used in pre-training. To this end, we compare pre-trained models in English and Netherlands Dutch to models trained on Flemish Dutch. Recent work [2]} has shown that low-resource languages can greatly benefit from higher-resource languages when they are more similar due to positive transfer, but cross-lingual representation learning degrades the performance on high-resource languages due to interference.
Furthermore, self-supervised pre-training has shown to improve robustness and reduce the degradation on out-of-domain data [3]}, [4]}. We show that simply augmenting the finetuning data leads to a strong speech recognition improvement in noisy and reverberated environments.
Finally, we investigate ASR improvements with the recent wav2vec 2.0 model [1]}, [2]} and study several pre-training and finetuning scenario's with an increasing amount of data, yielding substantial reduction of Word Error Rates (WER) compared to the baselines.
