While we are inspired by the HuBERT approach [1]}, our implementation and use differs in several aspects from the original work.
Our design decisions stem from a desire to reduce computational and data requirements.
Instead of building the initial clusters from MFCC features, we first train a Conformer [2]} model on a small amount of supervised out-of-domain data.
Once the model has been trained, the decoder portion of the network is discarded and only the encoder is kept.
We use the final layer of the encoder to generate embeddings for the untranscribed data.
After embedding the acoustic signal with the encoder, we cluster the embedded features using k-means.
As in [1]}, the clusters serve as labels for the unsupervised training.
We remove the convolutional waveform encoder and train a Conformer encoder from filterbank features instead.
During this step, there is no decoder involved.
The model is trained using frame-level cross-entropy using the clusters as targets.
After the encoder has been pretrained using the untranscribed data, a randomly initialized decoder is appended and the entire model is fine-tuned using the original transcribed audio.
