Moreover, compared to previous work that uses two models in tandem <cite>(Baevski et al., 2019b)</cite> , by using one model for both BERT pre-trainining and fine-tuning, our model provides an average relative WER reduction of 9%.