In all cases, the training recipe described in [1]} is followed. Upstream models are trained using a single NVidia A100 GPU. Experiments A and B take approximately 30 hours to train, whereas experiments C1 and C2 took each roughly 43 hours to train. We use the AdamW optimizer, with a batch of 24 utterances, for 200k iterations, whereas after 14k updates, the learning rate linearly decays from \(2\times 10^{-4}\)  to zero. As benchmarks, we use the original HuBERT Base model, its Robust version, and two compressed models: LightHuBERT and DistilHubert.
