@inproceedings{
xiong2024segtalker,
title={SegTalker: Segmentation-based Talking Face Generation with Mask-guided Local Editing},
author={Lingyu Xiong and Xize Cheng and Jintao Tan and Xianjia Wu and Xiandong Li and Lei Zhu and Fei Ma and Minglei Li and Huang Xu and Zhihui Hu},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=z0OEHZbT71}
}
Audio-driven talking face generation aims to synthesize video with lip movements synchronized to input audio. However, current generative techniques face challenges in preserving intricate regional textures (skin, teeth). To address the aforementioned challenges, we propose a novel framework called \textbf{SegTalker} to decouple lip movements and image textures by introducing segmentation as intermediate representation. Specifically, given the mask of image employed by a parsing network, we first leverage the speech to drive the mask and generate talking segmentation. Then we disentangle semantic regions of image into style codes using a mask-guided encoder. Ultimately, we inject the previously generated talking segmentation and style codes into a mask-guided StyleGAN to synthesize video frame. In this way, mostly of textures are fully preserved. Moreover, our approach can inherently achieve background separation and facilitate mask-guided facial local editing. In particular, by editing the mask and swapping the region textures from a given reference image (e.g. hair, lip, eyebrows), our approach enables facial editing seamlessly when generating talking face videos. Experiments demonstrate that our proposed approach can effectively preserve texture details and generate temporally consistent video while remaining competitive in lip synchronization. Quantitative results on the HDTF dataset illustrate the superior performance of our method over existing methods on most metrics.
SegTalker: Segmentation-based Talking Face Generation with Mask-guided Local Editing
Lingyu Xiong Xize Cheng Jintao Tan Xianjia Wu Xiandong Li Lei Zhu Fei Ma Minglei Li Huang Xu Zhihui Hu