Universal Speech Enhancement with Score-based Diffusion
Removing background noise from speech audio has been the subject of
considerable research and effort, especially in recent years due to the rise of
virtual communication and amateur sound recording. Yet background noise is not
the only unpleasant disturbance that can prevent intelligibility: reverb,
clipping, codec artifacts, problematic equalization, limited bandwidth, or
inconsistent loudness are equally disturbing and ubiquitous. In this work, we
propose to consider the task of speech enhancement as a holistic endeavor, and
present a universal speech enhancement system that tackles 55 different
distortions at the same time. Our approach consists of a generative model that
employs score-based diffusion, together with a multi-resolution conditioning
network that performs enhancement with mixture density networks. We show that
this approach significantly outperforms the state of the art in a subjective
test performed by expert listeners. We also show that it achieves competitive
objective scores with just 4-8 diffusion steps, despite not considering any
particular strategy for fast sampling. We hope that both our methodology and
technical contributions encourage researchers and practitioners to adopt a
universal approach to speech enhancement, possibly framing it as a generative
task.