@inproceedings{
lee2024daftgan,
title={{DAFT}-{GAN}: Dual Affine Transformation Generative Adversarial Network for Text-Guided Image Inpainting},
author={Jihoon Lee and Yunhong Min and Hwidong Kim and Sangtae Ahn},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=wqfOsjwryj}
}
In recent years, there has been a significant focus on research related to text-guided image inpainting, which holds a pivotal role in the domain of multimedia processing. This has resulted in notable enhancements in the quality and performance of the generated images. However, the task remains challenging due to several constraints, such as ensuring alignment between the generated images and the accompanying text, and maintaining consistency in distribution between corrupted and uncorrupted regions, for achieving natural and fine-grained image generation. To address these challenges, previous studies developed novel architectures, inpainting techniques, or objective functions but they still lack semantic consistency between the text and generated images. In this paper, thus, we propose a dual affine transformation generative adversarial network (DAFT-GAN) to maintain the semantic consistency for text-guided inpainting. DAFT-GAN integrates two affine transformation networks to combine text and image features gradually for each decoding block. The first affine transformation network leverages global features of the text to generate coarse results, while the second affine network utilizes attention mechanisms and spatial of the text to refine the coarse results. By connecting the features generated from these dual paths through residual connections in the subsequent block, the model retains information at each scale while enhancing the quality of the generated image. Moreover, we minimize information leakage of uncorrupted features for fine-grained image generation by encoding corrupted and uncorrupted regions of the masked image separately. Through extensive experiments, we observe that our proposed model outperforms the existing models in both qualitative and quantitative assessments with three benchmark datasets (MS-COCO, CUB, and Oxford) for text-guided image inpainting.
DAFT-GAN: Dual Affine Transformation Generative Adversarial Network for Text-Guided Image Inpainting
Jihoon Lee Yunhong Min Hwidong Kim Sangtae Ahn