A Survey on the New Generation of Deep Learning in Image Processing <s> B. STYLE TRANSFER <s> We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a per-pixel loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing perceptual loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results. <s> BIB001 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> B. STYLE TRANSFER <s> Rendering the semantic content of an image in different styles is a difficult image processing task. Arguably, a major limiting factor for previous approaches has been the lack of image representations that explicitly represent semantic information and, thus, allow to separate image content from style. Here we use image representations derived from Convolutional Neural Networks optimised for object recognition, which make high level image information explicit. We introduce A Neural Algorithm of Artistic Style that can separate and recombine the image content and style of natural images. The algorithm allows us to produce new images of high perceptual quality that combine the content of an arbitrary photograph with the appearance of numerous wellknown artworks. Our results provide new insights into the deep image representations learned by Convolutional Neural Networks and demonstrate their potential for high level image synthesis and manipulation. <s> BIB002 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> B. STYLE TRANSFER <s> We propose a method for transferring an arbitrary style to only a specific object in an image. Style transfer is the process of combining the content of an image and the style of another image into a new image. Our results show that the proposed method can realize style transfer to specific object. <s> BIB003
Rendering the semantic content of an image in different styles is a problematic image processing task BIB001 . Concretely, transferring the style from one image onto another can be considered a problem of texture transfer. In other words, the output must be semantically similar to the input despite changes in texture for style transfer. The drawback of previous traditional methods (without the support of deep learning) lies in the use of low-level features of target images to inform the texture transfer. Ideally, a style transfer algorithm should be able to extract the semantic image content from the target image and then inform a texture transfer procedure to render the semantic content of the target image in the style of the source image. Next, we introduce a framework of style transfer based on the deep CNN (see in Fig.25 ), concretely, how the generic feature representations learned by CNN can be used to independently process and manipulate the content and the style of natural images. As shown in Fig.25 , the framework consists of two components; one is image transform network f W , the other is loss network (here is VGG-16) that is used to define style/feature loss functions. For style transfer tasks, each input image x correspond to a content target y c and style target y s . In addition, the content target y c is the input image x, that is, y c = x. Generally, the image transform network is a deep ResNet with parameters W , which can transform x into output images y via the mapping y = f W (x). Further, to measure differences in content and style between y and y t = (y c , y s ), feature reconstruction loss l feat and style reconstruction loss l style can be defined. The right half of Fig.25 is a classical VGG-16 network, which is pre-trained on the ImageNet dataset. Further, The image transform network can be trained using stochastic gradient descent to minimize the following weighted combination of loss functions, where y (i) and y (i) t is the output image and target image of i-th layer for loss network, respectively. And l (i) is the loss function of i-th layer for loss network, that is, For i = 0, we have y y (0) = f W (x) and y t y t . In addition, for the definition of feature reconstruction loss and style reconstruction loss, please refer to the corresponding references. Unlike the previous style transfer with image transform network, let's introduce a simple style transfer algorithm BIB002 based on CNN in Fig.26 . First, the style image is passed through the Fig.26 network and its style representation on all layers included are computed and stored. The content (or input) image passed through the same network, and the content representation in one layer (such as the fourth layer's feature maps) is stored. Second, a random white noise image also passed through the same network, and its style features and content features can be computed. On each layer included in the style representation, style loss can obtain, and the content loss in one layer can also compute. Finally, the total loss is a linear combination of the content loss and the style loss, and its derivative concerning the pixel values of noise image can be computed using the BP algorithm. This gradient is used to iteratively update the noise image until it simultaneously matches the style features of the style image and the content features of the content image. Indeed, many CNNs frameworks can be used to extract abstract feature maps BIB003 . Therefore, the scheme of CNN can be used to transfer image style between arbitrary images is feasible.