Model setup We use an identical Conformer layer from SLAM [1]} and mSLAM [2]}, in which the model dimension is 1024, feedforward hidden dimension is 4096, convolution kernel size is 5 and the number of attention heads is 8. The Transformer layers in the decoder share the same setting as Conformer layer in terms of model dimension, hidden dimension and attention heads but we set dropout to \(0.1\)  for the Transformer layers rather than the default 0 in the Conformer layers. We use the same learning schedule as Transformer during pre-training and fine-tuning but warmup steps are set to \(40k\)  and \(16k\)  respectively. The Adam optimizer [3]} is applied to pre-traing with learning rate as 3 while AdamW [4]} is used as fine-tuning optimizer with weight decay rate as \(0.01\) .
