@inproceedings{jain-etal-2024-multi,
    title = "Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition",
    author = "Jain, Yash  and
      Chan, David M.  and
      Dheram, Pranav  and
      Khare, Aparna  and
      Shonibare, Olabanji  and
      Ravichandran, Venkatesh  and
      Ghosh, Shalini",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1045",
    pages = "11969--11980",
    abstract = "Recent advances in machine learning have demonstrated that multi-modal pre-training can improve automatic speech recognition (ASR) performance compared to randomly initialized models, even when models are fine-tuned on uni-modal tasks. Existing multi-modal pre-training methods for the ASR task have primarily focused on single-stage pre-training where a single unsupervised task is used for pre-training followed by fine-tuning on the downstream task. In this work, we introduce a novel method combining multi-modal and multi-task unsupervised pre-training with a translation-based supervised mid-training approach. We empirically demonstrate that such a multi-stage approach leads to relative word error rate (WER) improvements of up to 38.45{\%} over baselines on both Librispeech and SUPERB. Additionally, we share several important findings for choosing pre-training methods and datasets.",
}

Recent advances in machine learning have demonstrated that multi-modal pre-training can improve automatic speech recognition (ASR) performance compared to randomly initialized models, even when models are fine-tuned on uni-modal tasks. Existing multi-modal pre-training methods for the ASR task have primarily focused on single-stage pre-training where a single unsupervised task is used for pre-training followed by fine-tuning on the downstream task. In this work, we introduce a novel method combining multi-modal and multi-task unsupervised pre-training with a translation-based supervised mid-training approach. We empirically demonstrate that such a multi-stage approach leads to relative word error rate (WER) improvements of up to 38.45{\%} over baselines on both Librispeech and SUPERB. Additionally, we share several important findings for choosing pre-training methods and datasets.
Jain, Yash Chan, David M. Dheram, Pranav Khare, Aparna Shonibare, Olabanji Ravich ran, Venkatesh Ghosh, Shalini
Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition