We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a gumbel softmax or online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition. Learning discrete representations of speech has gathered much recent interest (Versteegh et al., 2016; Dunbar et al., 2019) . A popular approach to discover discrete units is via autoencoding (Tjandra et al., 2019; Eloff et al., 2019; Chorowski et al., 2019) sometimes coupled with an autoregressive model . Another line of research is to learn continuous speech representations in a self-supervised way via predicting context information (Chung & Glass, 2018; van den Oord et al., 2018; Schneider et al., 2019) . In this paper, we combine these two lines of research by learning discrete representations of speech via a context prediction task instead of reconstructing the input. This enables us to directly apply well performing NLP algorithms to speech data ( Figure 1a ). The vq-wav2vec encoder maps raw audio (X ) to a dense representation (Z) which is quantized (q) toẐ and aggregated into context representations (C); training requires future time step prediction. (b) Acoustic models are trained by quantizing the raw audio with vq-wav2vec, then applying BERT to the discretized sequence and feeding the resulting representations into the acoustic model to output transcriptions. Our new discretization algorithm, vq-wav2vec, learns discrete representations of fixed length segments of audio signal by utilizing the wav2vec loss and architecture (Schneider et al, 2019; §2) . To choose the discrete variables, we consider a Gumbel-Softmax approach (Jang et al., 2016) as well as online k-means clustering, similar to VQ-VAE (Oord et al., 2017; Eloff et al., 2019; §3) . We then train a Deep Bidirectional Transformer (BERT; Devlin et al., 2018; on the discretized unlabeled speech data and input these representations to a standard acoustic model (Figure 1b; §4) . Our experiments show that BERT representations perform better than log-mel filterbank inputs as well as dense wav2vec representations on both TIMIT and WSJ benchmarks. Discretization of audio enables the direct application of a whole host of algorithms from the NLP literature to speech data. For example, we show that a standard sequence to sequence model from the NLP literature can be used to perform speech recognition over discrete audio tokens ( §5, §6). vq-wav2vec is a self-supervised algorithm that quantizes unlabeled audio data which makes it amenable to algorithms requiring discrete data. This approach improves the state of the art on the WSJ and TIMIT benchmarks by leveraging BERT pre-training. In future work, we plan to apply other algorithms requiring discrete inputs to audio data and to explore self-supervised pre-training algorithms which mask part of the continuous audio input. Another future work avenue is to finetune the pre-trained model to output transcriptions instead of feeding the pre-trained features to a custom ASR model.
Learn how to quantize speech signal and apply algorithms requiring discrete inputs to audio data such as BERT.
{"src": "scitldr-AIC_train", "paper_id": "rylwJxrYDS"}