In this work we use the HuBERT approach to unsupervised learning to pretrain the encoder of a Conformer model for the purposes of domain adaptation.
The model is then fine-tuned on out-of-domain supervised data.
We compare unsupervised learning with semi-supervised learning, both separately and in combination.
We also have access to large amounts of text data in the new domain.
Despite the advance of techniques like shallow fusion, deep fusion [1]}, cold fusion [2]}, and internal language model estimation [3]}, there is a limit to how much an autoregressive model can gain from additional language model data.
We experiment with utilizing the encoder from the Conformer model as a CTC model in a non-autoregressive framework in order to take full advantage of the additional text data.
Our contributions include:
