EPG2S: Speech Generation and Speech Enhancement based on Electropalatography and Audio Signals using Multimodal Learning
Speech generation and enhancement based on articulatory movements facilitate
communication when the scope of verbal communication is absent, e.g., in
patients who have lost the ability to speak. Although various techniques have
been proposed to this end, electropalatography (EPG), which is a monitoring
technique that records contact between the tongue and hard palate during
speech, has not been adequately explored. Herein, we propose a novel multimodal
EPG-to-speech (EPG2S) system that utilizes EPG and speech signals for speech
generation and enhancement. Different fusion strategies based on multiple
combinations of EPG and noisy speech signals are examined, and the viability of
the proposed method is investigated. Experimental results indicate that EPG2S
achieves desirable speech generation outcomes based solely on EPG signals.
Further, the addition of noisy speech signals is observed to improve quality
and intelligibility. Additionally, EPG2S is observed to achieve high-quality
speech enhancement based solely on audio signals, with the addition of EPG
signals further improving the performance. The late fusion strategy is deemed
to be the most effective approach for simultaneous speech generation and
enhancement.