Artistic style transfer is an image synthesis problem where the content of an image is reproduced with the style of another. Recent works show that a visually appealing style transfer can be achieved by using the hidden activations of a pretrained convolutional neural network. However, existing methods either apply (i) an optimization procedure that works for any style image but is very expensive, or (ii) an efficient feedforward network that only allows a limited number of trained styles. In this work we propose a simpler optimization objective based on local matching that combines the content structure and style textures in a single layer of the pretrained network. We show that our objective has desirable properties such as a simpler optimization landscape, intuitive parameter tuning, and consistent frame-by-frame performance on video. Furthermore, we use 80,000 natural images and 80,000 paintings to train an inverse network that approximates the result of the optimization. This results in a procedure for artistic style transfer that is efficient but also allows arbitrary content and style images.
In comparison to existing style transfer approaches, we propose a method for directly constructing the target activations for a single layer in a pretrained CNN. Like Li and Wand @cite_5 , we make use of a criteria for finding best matching patches in the activation space. However, our method is able to directly construct the entire activation target. This deterministic procedure allows us to easily adapt to video, without the need to rely on optical flow for fixing consistency issues. In addition to optimization, we propose a feed-forward style transfer procedure by inverting the pretrained CNN. Unlike existing feed-forward style transfer approaches, our method is not limited to specifically trained styles and can easily adapt to arbitary content and style images. Unlike existing CNN inversion methods, our method of training does not use a pixel-level loss, but instead uses a loss on the activations. By using a particular training setup (see ), this inverse network is even able to invert activations that are outside the usual range of the CNN activiations.
@cite_5
2275363859
Artistic style transfer is an image synthesis problem where the content of an image is reproduced with the style of another. Recent works show that a visually appealing style transfer can be achieved by using the hidden activations of a pretrained convolutional neural network. However, existing methods either apply (i) an optimization procedure that works for any style image but is very expensive, or (ii) an efficient feedforward network that only allows a limited number of trained styles. In this work we propose a simpler optimization objective based on local matching that combines the content structure and style textures in a single layer of the pretrained network. We show that our objective has desirable properties such as a simpler optimization landscape, intuitive parameter tuning, and consistent frame-by-frame performance on video. Furthermore, we use 80,000 natural images and 80,000 paintings to train an inverse network that approximates the result of the optimization. This results in a procedure for artistic style transfer that is efficient but also allows arbitrary content and style images.