Most SSL models rely on learning speech representations from clean readings of audiobook data, such as LibriSpeech [1]} or LibriLight [2]}. Even though these models can learn essential characteristics from this type of speech, real-world deployment often involves noisy speech utterances which degrade system performance. Data augmentation has commonly been applied to improve the robustness to unseen conditions [3]}. Here, we propose to perform an online contamination of the data during the distillation process. In particular, the student model receives the noisy data as input, but the network's target is to reconstruct the clean features of the teacher model. At training time, given a batch of clean speech utterances, we uniformly sample one action to be applied to each utterance in the batch:
