Our multi-task multilingual speech-text pre-training method (Mu\(^{2}\) SLAM) is based on an encoder-decoder backbone model. For speech inputs, we follow SLAM [1]} and mSLAM [2]} to convert an acoustic feature sequence of 80-dimensional log Mel spectrograms into a sequence of latent speech representations via a CNN block. The CNN block consisting of two 2D-convolutional layers with strides (2, 2) also acts as a sub-sampling mechanism with a 4x reduction in the sequence length dimension. A subsequent linear projection layer is used to map the dimension of the latent speech representations to that of the encoder stack, we denote the speech representations as \(\mathbf {S}\) . The text input \(\mathbf {t}\)  simply goes through a token embedding layer to be transformed as a sequence of embeddings. To specify the language and modality, we add language and modality embeddings to word embeddings or speech representations \(\mathbf {S}\)  in addition to the conventional positional embeddings. The speech and text representations are then feed into a shared multi-modal encoder-decoder model. We prefer a deep encoder with 24 Conformer layers [3]} (a
similar encoder as mSLAM) and a shallow decoder with 6 Transformer layers [4]}, which favors faster inference while maintaining competitive quality [5]}.
