Different from <cite>(Baevski et al., 2019b)</cite> where the a BERT-like model is trained with the masked language model loss, frozen, and then used as a feature extractor in tandem with a final fully supervised convolutional ASR model (Collobert et al., 2016) , in this work, our "Discrete BERT" approach achieves an average relative Word Error Rate (WER) reduction of 9% by pre-training and fine-tuning the same BERT model using a Connectionist Temporal Classification (Graves et al.) loss.