NASTAR: Noise Adaptive Speech Enhancement with Target-Conditional Resampling
For deep learning-based speech enhancement (SE) systems, the training-test
acoustic mismatch can cause notable performance degradation. To address the
mismatch issue, numerous noise adaptation strategies have been derived. In this
paper, we propose a novel method, called noise adaptive speech enhancement with
target-conditional resampling (NASTAR), which reduces mismatches with only one
sample (one-shot) of noisy speech in the target environment. NASTAR uses a
feedback mechanism to simulate adaptive training data via a noise extractor and
a retrieval model. The noise extractor estimates the target noise from the
noisy speech, called pseudo-noise. The noise retrieval model retrieves relevant
noise samples from a pool of noise signals according to the noisy speech,
called relevant-cohort. The pseudo-noise and the relevant-cohort set are
jointly sampled and mixed with the source speech corpus to prepare simulated
training data for noise adaptation. Experimental results show that NASTAR can
effectively use one noisy speech sample to adapt an SE model to a target
condition. Moreover, both the noise extractor and the noise retrieval model
contribute to model adaptation. To our best knowledge, NASTAR is the first work
to perform one-shot noise adaptation through noise extraction and retrieval.