As the ASR backend, we use the large size Conformer Transducer refereed as Conformer(L) in [1]} which has 17 Conformer blocks of 512-dim 8-head dot-product attention, 512-dim 32-frame kernel 1d convolution, 2048-dim feedforward module, and a 640-dim long short-term memory (LSTM) decoder. The input feature was 80-dim filterbanks features computed from a 25 ms window with a stride of 10 ms. We used SpecAugment [2]} with mask parameter (\(F = 27\) ), and ten time masks with maximum time-mask ratio (\(p_S = 0.05\) ). All transciptions were tokenized with a word piece model with a 1,024 vocabulary built from LibriSpeech 960h. The ASR loss \(\mathcal {L}^{\mbox{\tiny {ASR}}}\)  was the recurrent neural network transducer (RNN-T) loss described in [3]}.
