Although these works have achieved impressive performance, they only consider partial aspects of the unified models in speech and text. First, except for SLAM and mSLAM [1]}, [2]}, most of them merely focus on speech-related tasks by taking text data as auxiliary inputs while ignoring the evaluation on text-related benchmarks, which leaves us unknown to gauge the effect of interference and capacity dilution. Second, there are few studies investigating multilingual modeling with both speech and text [2]}, [4]}, which limits them in leveraging cross-lingual transfer to enrich the speech and text joint representations. Third, multi-task learning has demonstrated the effectiveness of inductive transfer to improve model generalization, yet it is understudied in speech-text pre-training [5]}, [6]}, [4]} where they explicitly differentiate the utilization of labeled data in pre-training by introducing customized networks and losses.
Fourth, it is essential for prior speech-text models to design modality-specific blocks and losses to yield high performance [2]}, [4]}, [5]}, [6]}, [12]} which somewhat violates the principle of the unified models by using one model for all tasks, thus undermining the language and modality transfer to learn general speech and text shared representations.
