A Survey on the New Generation of Deep Learning in Image Processing <s> D. SUPER RESOLUTION <s> In this paper we introduce a generative parametric model capable of producing high quality samples of natural images. Our approach uses a cascade of convolutional networks within a Laplacian pyramid framework to generate images in a coarse-to-fine fashion. At each level of the pyramid, a separate generative convnet model is trained using the Generative Adversarial Nets (GAN) approach [11]. Samples drawn from our model are of significantly higher quality than alternate approaches. In a quantitative assessment by human evaluators, our CIFAR10 samples were mistaken for real images around 40% of the time, compared to 10% for samples drawn from a GAN baseline model. We also show samples from models trained on the higher resolution images of the LSUN scene dataset. <s> BIB001 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> D. SUPER RESOLUTION <s> We propose a deep learning method for single image super-resolution (SR). Our method directly learns an end-to-end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art restoration quality, and achieves fast speed for practical on-line usage. We explore different network structures and parameter settings to achieve trade-offs between performance and speed. Moreover, we extend our network to cope with three color channels simultaneously, and show better overall reconstruction quality. <s> BIB002 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> D. SUPER RESOLUTION <s> Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method. <s> BIB003 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> D. SUPER RESOLUTION <s> We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2. We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CelebA dataset. <s> BIB004
Single image super-resolution (SR) aims to reconstruct a high-resolution (HR) image from one single low resolution (LR) input image. As the pioneer CNN model for SR, SRCNN predicts the non-linear LR-to-HR mapping function via a FCN and significantly outperforms classical non-deep learning methods BIB002 . However, one fundamental problem remains unsolved mainly: how does one recover the finer texture details when super-resolution at large up-scaling factors? In recent years, GANs can provide a robust framework for generating plausible-looking natural images with high perceptual quality. Naturally, SRGAN BIB003 (see in Fig.27 ), which is a GAN-based network optimized for a new perceptual loss, can recover photo-realistic textures from heavily down-sampled images of public benchmarks. Further, a discriminator network D which can optimize in an alternating manner along with G to solve the following adversarial minmax problem: Generally, it allows training a generative model G to fool a differentiable discriminator D that is trained to distinguish SR (or generated) images from real images. Without loss of generality, for training samples I HR n with corresponding I LR n , n = 1, 2, · · · , N . θ G can be solved by the following  where loss SR is perceptual loss function, it can be divided into content loss and adversarial loss. Moreover, the generator model is the core of SRGAN, which illustrated in Fig.27 are residual blocks with an identical layout. Finally, extensive quantitative and qualitative evaluations on benchmark datasets show that SRGAN can recover the finer texture details when super-resolution at large up-scaling factors. Besides, based on the idea of CGAN, LAPGAN can generates high-quality pictures based on GAN to solve the problem of poor quality of data generated by GAN BIB001 . The framework of LAPGAN is illustrated in Fig.28 . Starting with a 64 × 64 input image I from training set, we take I = I 0 and blur and downsample it by a factor of two to produce I 1 , then we can up sample I 1 by a factor of two, giving a low-pass version l 0 of I 0 . Further, we compute high-pass h 0 = I 0 l 0 , h 0 is input to the discriminative model D 0 that computes the probability of it being real vs generated. The same procedure is repeated at scales 1 and 2, using I 1 and I 2 . At level 3, I 3 is an 8 × 8 image, simple enough to be modeled directly with a standard GANs G 3 and D 3 . Note that the models at each level can be trained separately. Moreover, the most significant difference between the PGGAN BIB004 and SRGAN and LAPGAN are that the structure of the latter two is fixed, but the structure of PGGAN is continuously changing as the training progresses. The most significant benefit of this is that most of the iterations of PGGAN are completed at a lower resolution, and the training speed is 2-6 times higher than that of traditional GANs. The main advantage of PGGAN is that it can generate high-quality samples. There are many HD pictures in our daily life, the application value of PGGAN is quite a significant promising.