In this work we propose Pixel Content Encoders (PCE), a light-weight image inpainting model, capable of generating novel con-tent for large missing regions in images. Unlike previously presented convolutional neural network based models, our PCE model has an order of magnitude fewer trainable parameters. Moreover, by incorporating dilated convolutions we are able to preserve fine grained spatial information, achieving state-of-the-art performance on benchmark datasets of natural images and paintings. Besides image inpainting, we show that without changing the architecture, PCE can be used for image extrapolation, generating novel content beyond existing image boundaries.
Ablation studies presented in a number of works on inpainting have shown that the structural (e.g., L1 or L2) loss results in blurry images @cite_13 @cite_14 @cite_12 . Nonetheless, these blurry images do accurately capture the coarse structure, i.e., the low spatial frequencies. This matches an observation by @cite_21 , who stated that if the structural loss captures the low spatial frequencies, the GAN loss can be tailored to focus on the high spatial frequencies (the details). Specifically, introduced PatchGAN, a GAN which focuses on the structure in local patches, relying on the structural loss to ensure correctness of the global structure. PatchGAN, produces a judgement for @math patches, where @math can be much smaller than the whole image. When @math is smaller than the image, PatchGAN is applied convolutionally and the judgements are averaged to produce a single outcome.
@cite_14 @cite_13 @cite_21 @cite_12
2523714292 2963470893 2791184993 2765732129
Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method. Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method. Area of image inpainting over relatively large missing regions recently advanced substantially through adaptation of dedicated deep neural networks. However, current network solutions still introduce undesired artifacts and noise to the repaired regions. We present an image inpainting method that is based on the celebrated generative adversarial network (GAN) framework. The proposed PGGAN method includes a discriminator network that combines a global GAN (G-GAN) architecture with a patchGAN approach. PGGAN first shares network layers between G-GAN and patchGAN, then splits paths to produce two adversarial losses that feed the generator network in order to capture both local continuity of image texture and pervasive global features in images. The proposed framework is evaluated extensively, and the results including comparison to recent state-of-the-art demonstrate that it achieves considerable improvements on both visual and quantitative evaluations. In recent years, we have witnessed the great success of deep learning on various problems both in low and high-level computer visions. The low-level vision problems, including inpainting, deblurring, denoising, super-resolution, and so on, are highly anticipated to occur in machine vision and image processing. Many deep learning based methods have been proposed to solve low-level vision problems. Most researches treat these problems independently; however, most of the time they appear concurrently. Motivated by the success of generative model in the field of image generation, we develop a deep cascade of neural networks to solve the inpainting, deblurring, denoising problems at the same time. Our model contains two networks: inpainting GAN and deblurring-denoising network. Inpainting GAN generates the coarse patches to fill the lost part in damaged image, and the deblurring-denoising network, stacked by a convolutional auto-encoder, will further refine them. Unlike other methods that handle each problem separately, our method jointly optimizes the two sub-networks. Because GAN training is not only unstable but also difficult, we adopt the Wasserstein distance as the loss function of the inpainting GAN and propose a gradual training strategy. Learning from the idea of residual learning, we utilize skip connections to pass image details from input to reconstruction layer. Experimental results have demonstrated that the proposed model can achieve state-of-the-art performance. Through the experiments, we also demonstrated the effectiveness of the cascade architecture.