Our work builds on the recently proposed work in <cite>(Baevski et al., 2019b)</cite> where audio is quantized using a contrastive loss, then features learned on top by a BERT model (Devlin et al., 2018) .