Compared to the two-model tandem system proposed in <cite>(Baevski et al., 2019b)</cite> , which uses a the discrete BERT features to train another ASR system from scratch, our discrete BERT model provides an average of 13% and 6% of WER reduction on clean and other subsets respectively, by pre-training and fine-tuning the same BERT model on the 10h labeled set.