During unsupervised pretraining, we use frame-level cross entropy to train the Conformer encoder with the k-means clusters as targets.
The encoder is trained for a maximum of 40 epochs.
The minibatch size is 128 utterances, corresponding to approximately 500 seconds of audio.
We use SpecAugment [1]} during both supervised and unsupervised training.
