Speech-text Pre-training
Pre-training methods have dominated research and industry fields due to their superior capabilities of exploiting unlabeled data. Particularly, in NLP and speech, a lot of representatives, such as BERT [1]}, XLNET [2]}, T5 [3]}, MASS [4]}, wav2vec [5]}, Hubert [6]} and so on, come out to improve mono-modal model performance. In recent days, the research community has started to move towards speech-text joint training, aiming to learn the shared representation space between speech and text. There are a lot of work along this research line which can be roughly categorized
into, encoder-only pre-training [7]}, [8]}, [9]}, encoder-decoder pre-training [10]}, [11]}, [12]}, [13]}, [14]}, [15]}. Our approach Mu\(^{2}\) SLAM adopts an encoder-decoder backbone model but our model design minimizes the utilization of modality-specific blocks only with a CNN block used to extract speech representations, which dramatically
simplify the cross-modal model architecture and enforce the representation sharing from the beginning. Among them, the most related Maestro [11]} also incorporates ASR, AST and MT data into their method, however, their model training has to rely on a pre-trained mSLAM as initialization and applies a duration model to over-sample the text which is not activated during fine-tuning. In contrast, Mu\(^{2}\) SLAM pre-trains the model from scratch which can be applied in the downstream tasks without wasting any parameter. We also verify our models on text-related benchmarks while they just focus on speech tasks. In addition, a text-to-speech loss is also introduced in pre-training which endows our model with the ability of speech generation. We leave it as the future exploration.
