Multilingual Pre-training The great success of multilingual text pre-training like mBERT [1]}, XLM-R [2]} and mT5 [3]} incentivizes the speech research to move toward multilingual modeling and pre-training [4]}, [5]}, [6]}, [7]}, which benefits from the cross-lingual transfer for learning joint representations across massive amounts of data across multiple languages [8]}, [9]}. Our approach is inspired by this research direction by involving multilingual speech and text spanning over 100 languages.
