@inproceedings{
lai2023training,
title={Training Speech Recognition Models to Follow Instructions},
author={Cheng-I Lai and Zhiyun Lu and Liangliang Cao and Ruoming Pang},
booktitle={NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following},
year={2023},
url={https://openreview.net/forum?id=wYLASkYFJU}
}
Conventional end-to-end Automatic Speech Recognition (ASR) models primarily focus on exact transcription tasks, lacking flexibility for nuanced user interactions. In this paper, we train a speech recognition model to follow a diverse set of free-form text instructions for a multitude of speech recognition tasks -- ranging from simple transcript manipulation to summarization. We emphasize that even without pre-trained LLMs or speech modules, a Listen-Attend-Spell model trained from scratch on Librispeech understands and executes instructions with high fidelity. This preliminary findings highlight the potential of instruction-following training to advance speech foundation models.
Training Speech Recognition Models to Follow Instructions
Cheng-I Lai Zhiyun Lu Liangliang Cao Ruoming Pang