We propose a novel generative adversarial network for visual attributes manipulation (ManiGAN), which is able to semantically modify the visual attributes of given images using natural language descriptions. The key to our method is to design a novel co-attention module to combine text and image information rather than simply concatenating two features along the channel direction. Also, a detail correction module is proposed to rectify mismatched attributes of the synthetic image, and to reconstruct text-unrelated contents. Finally, we propose a new metric for evaluating manipulation results, in terms of both the generation of text-related attributes and the reconstruction of text-unrelated contents. Extensive experiments on benchmark datasets demonstrate the advantages of our proposed method, regarding the effectiveness of image manipulation and the capability of generating high-quality results. Image manipulation refers to the task of changing various aspects of given images from low-level colour or texture Gatys et al., 2016) to high-level semantics (Zhu et al., 2016) , and has numerous potential applications in video games, image editing, and computer-aided design. Recently, with the development of deep learning and generative models, automatic image manipulation becomes possible, including image inpainting (Iizuka et al., 2016; Pathak et al., 2016) , image colourisation , style transfer (Gatys et al., 2016; Johnson et al., 2016) , and domain or attribute translation (Lample et al., 2017; . However, all the above works mainly focus on specific tasks, and only few studies (Dong et al., 2017; Nam et al., 2018) concentrate on more general and user-friendly image manipulation by using natural language descriptions. Also, as shown in Fig.1 , current state-of-the-art methods can only generate low-quality images and fail to effectively manipulate given images on more complicated datasets, such as COCO (Lin et al., 2014) . The less effective performance is mainly because (1) simply concatenating text and image cross-domain features along the channel direction, the model may fail to precisely correlate words and corresponding visual attributes, and thus cannot modified specific attributes required in the text, and (2) conditioned only on a global sentence vector, current state-of-the-art methods lack important fine-grained information at the word-level, which prevents an effective manipulation using natural language descriptions. In this paper, we aim to manipulate given images using natural language descriptions. In particular, we focus on modifying visual attributes (e.g., category, texture, colour, and background) of input images by providing texts that describe desired attributes. To achieve this, we propose a novel generative adversarial network for visual attributes manipulation (ManiGAN), which allows to effectively manipulate given images using natural language descriptions and to produce high-quality results. The contribution of our proposed method is fourfold: (1) instead of simply concatenating hidden features generated from a natural language description and image features encoded from the input image along the channel direction, we propose a novel co-attention module where both features can collaborate to reconstruct the input image and also keep the synthetic result semantically aligned with the given text description, (2) a detail correction module (DCM) is introduced to rectify mismatched attributes, and to reconstruct text-unrelated contents existing in the input image, (3) a new metric is proposed, which can appropriately reflect the generation of text-related visual attributes and the reconstruction of text-unrelated contents involved in the image manipulation, and (4) extensive experiments on the CUB (Wah et al., 2011) and COCO (Lin et al., 2014) Figure 1: Examples of image manipulation using natural language descriptions. Current state-of-theart methods only generate low-quality images, and fail to do manipulation on COCO. In contrast, our method allows the input images to be manipulated accurately corresponding to the given text descriptions while preserving text-unrelated contents. to demonstrate the superiority of our model, which outperforms existing state-of-the-art methods both qualitatively and quantitatively. We have proposed a novel generative adversarial network for visual attributes manipulation, called ManiGAN, which can semantically manipulate the input images using natural language descriptions. Two novel components are proposed in our model: (1) the co-attention module enables cooperation between hidden features and image features where both features can collaborate to reconstruct the input image and also keep the synthetic result semantically aligned with the given text description, and (2) the detail correction module can rectify mismatched visual attributes of the synthetic result, and also reconstruct text-unrelated contents existing in the input image. Extensive experimental results demonstrate the superiority of our proposed method, in terms of both the effectiveness of image manipulation and the capability of generating high-quality results.
We propose a novel method to manipulate given images using natural language descriptions.
{"src": "scitldr-AIC_train", "paper_id": "HJl6tC4KwB"}