Image inpainting is the task of filling-in missing regions of a damaged or incomplete image. In this work we tackle this problem not only by using the available visual data but also by incorporating image semantics through the use of generative models. Our contribution is twofold: First, we learn a data latent space by training an improved version of the Wasserstein generative adversarial network, for which we incorporate a new generator and discriminator architecture. Second, the learned semantic information is combined with a new optimization loss for inpainting whose minimization infers the missing content conditioned by the available data. It takes into account powerful contextual and perceptual content inherent in the image itself. The benefits include the ability to recover large regions by accumulating semantic information even it is not fully present in the damaged image. Experiments show that the presented method obtains qualitative and quantitative top-tier results in different experimental situations and also achieves accurate photo-realism comparable to state-of-the-art works.
Most inpainting methods found in the literature can be classified into two groups: model-based approaches and deep learning approaches. In the former, two main groups can be distinguished: local and non-local methods. In local methods, also denoted as geometry-oriented methods, images are modeled as functions with some degree of smoothness. @cite_1 @cite_16 @cite_6 @cite_43 @cite_18 . These methods show good performance in propagating smooth level lines or gradients, but fail in the presence of texture or for large missing regions. Non-local methods (also called exemplar- or patch-based) exploit the self-similarity prior by directly sampling the desired texture to perform the synthesis @cite_25 @cite_7 @cite_35 @cite_23 @cite_0 @cite_8 @cite_20 @cite_26 @cite_9 . They provide impressive results in inpainting textures and repetitive structures even in the case of large holes. However, both type of methods use redundancy of the incomplete input image: smoothness priors in the case of geometry-based and self-similarity principles in the non-local or patch-based ones. Figures (b) and (c) illustrate the inpainting results (the inpaining hole is shown in (a)) using a local method (in particular @cite_43 ) and the non-local method @cite_44 , respectively. As expected, the use of image semantics improve the results, as shown in (d).
@cite_35 @cite_18 @cite_26 @cite_7 @cite_8 @cite_9 @cite_1 @cite_6 @cite_0 @cite_43 @cite_44 @cite_23 @cite_16 @cite_25 @cite_20
2963540914 2784790939 2791184993 1999905919
Recent deep learning based approaches have shown promising results for the challenging task of inpainting large missing regions in an image. These methods can generate visually plausible image structures and textures, but often create distorted structures or blurry textures inconsistent with surrounding areas. This is mainly due to ineffectiveness of convolutional neural networks in explicitly borrowing or copying information from distant spatial locations. On the other hand, traditional texture and patch synthesis approaches are particularly suitable when it needs to borrow textures from the surrounding regions. Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions. The model is a feedforward, fully convolutional neural network which can process images with multiple holes at arbitrary locations and with variable sizes during the test time. Experiments on multiple datasets including faces (CelebA, CelebA-HQ), textures (DTD) and natural images (ImageNet, Places2) demonstrate that our proposed approach generates higher-quality inpainting results than existing ones. Code, demo and models are available at: https: github.com JiahuiYu generative_inpainting. Recent deep learning based approaches have shown promising results on image inpainting for the challenging task of filling in large missing regions in an image. These methods can generate visually plausible image structures and textures, but often create distorted structures or blurry textures inconsistent with surrounding areas. This is mainly due to ineffectiveness of convolutional neural networks in explicitly borrowing or copying information from distant spatial locations. On the other hand, traditional texture and patch synthesis approaches are particularly suitable when it needs to borrow textures from the surrounding regions. Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions. The model is a feed-forward, fully convolutional neural network which can process images with multiple holes at arbitrary locations and with variable sizes during the test time. Experiments on multiple datasets including faces, textures and natural images demonstrate that the proposed approach generates higher-quality inpainting results than existing ones. Code and trained models will be released. Area of image inpainting over relatively large missing regions recently advanced substantially through adaptation of dedicated deep neural networks. However, current network solutions still introduce undesired artifacts and noise to the repaired regions. We present an image inpainting method that is based on the celebrated generative adversarial network (GAN) framework. The proposed PGGAN method includes a discriminator network that combines a global GAN (G-GAN) architecture with a patchGAN approach. PGGAN first shares network layers between G-GAN and patchGAN, then splits paths to produce two adversarial losses that feed the generator network in order to capture both local continuity of image texture and pervasive global features in images. The proposed framework is evaluated extensively, and the results including comparison to recent state-of-the-art demonstrate that it achieves considerable improvements on both visual and quantitative evaluations. This paper describes a novel inpainting algorithm that is capable of filling in holes in overlapping texture and cartoon image layers. This algorithm is a direct extension of a recently developed sparse-representation-based image decomposition method called MCA (morphological component analysis), designed for the separation of linearly combined texture and cartoon layers in a given image (see [J.-L. Starck, M. Elad, D.L. Donoho, Image decomposition via the combination of sparse representations and a variational approach, IEEE Trans. Image Process. (2004), in press] and [J.-L. Starck, M. Elad, D.L. Donoho, Redundant multiscale transforms and their application for morphological component analysis, Adv. Imag. Electron Phys. (2004) 132]). In this extension, missing pixels fit naturally into the separation framework, producing separate layers as a by-product of the inpainting process. As opposed to the inpainting system proposed by , where image decomposition and filling-in stages were separated as two blocks in an overall system, the new approach considers separation, hole-filling, and denoising as one unified task. We demonstrate the performance of the new approach via several examples.