Artistic style transfer is the problem of synthesizing an image with content similar to a given image and style similar to another. Although recent feed-forward neural networks can generate stylized images in real-time, these models produce a single stylization given a pair of style/content images, and the user doesn't have control over the synthesized output. Moreover, the style transfer depends on the hyper-parameters of the model with varying ``optimum" for different input images. Therefore, if the stylized output is not appealing to the user, she/he has to try multiple models or retrain one with different hyper-parameters to get a favorite stylization. In this paper, we address these issues by proposing a novel method which allows adjustment of crucial hyper-parameters, after the training and in real-time, through a set of manually adjustable parameters. These parameters enable the user to modify the synthesized outputs from the same pair of style/content images, in search of a favorite stylized image. Our quantitative and qualitative experiments indicate how adjusting these parameters is comparable to retraining the model with different hyper-parameters. We also demonstrate how these parameters can be randomized to generate results which are diverse but still very similar in style and content. Style transfer is a long-standing problem in computer vision with the goal of synthesizing new images by combining the content of one image with the style of another BID8 BID12 BID0 . Recently, neural style transfer techniques BID9 BID15 BID11 BID20 BID19 showed that the correlation between the features extracted from the trained deep neural networks is quite effective on capturing the visual styles and content that can be used for generating images similar in style and content. However, since the definition of similarity is inherently vague, the objective of style transfer is not well defined and one can imagine multiple stylized images from the same pair of content/style images.Existing real-time style transfer methods generate only one stylization for a given content/style pair and while the stylizations of different methods usually look distinct BID27 BID13 , it is not possible to say that one stylization is better in all contexts since people react differently to images based on their background and situation. Hence, to get favored stylizations users must try different methods that is not satisfactory. It is more desirable to have a single model which can generate diverse results, but still similar in style and content, in real-time, by adjusting some input parameters.One other issue with the current methods is their high sensitivity to the hyper-parameters. More specifically, current real-time style transfer methods minimize a weighted sum of losses from different layers of a pre-trained image classification model BID15 BID13 (check Sec 3 for details) and different weight sets can result into very different styles (Figure 6) . However, one can only observe the effect of these weights in the final stylization by fully retraining the model with the new set of weights. Considering the fact that the "optimal" set of weights can be different for any pair of style/content ( Figure 3 ) and also the fact that this "optimal" truly doesn't exist (since the goodness of the output is a personal choice) retraining the models over and over until the desired result is generated is not practical. Our main contribution in this paper is a novel method which allows adjustment of each loss layer's contribution in feed-forward style transfer networks, in real-time and after training. This capability allows the users to adjust the stylized output to find the favorite stylization by changing input parameters and without retraining the stylization model. We also show how randomizing these parameters plus some noise added to the content image can result in very different stylizations from the same pair of style/content image.Our method can be expanded in numerous ways e.g. applying it to multi-style transfer methods such as BID11 , applying the same parametrization technique to randomize the correlation loss between the features of each layer and finally using different loss functions and pre-trained networks for computing the loss to randomize the outputs even further. One other interesting future direction is to apply the same "loss adjustment after training" technique for other classic computer vision and deep learning tasks. Style transfer is not the only task in which modifying the hyper-parameters can greatly affect the predicted results and it would be rather interesting to try this method for adjusting the hyper-parameters in similar problems. Convolution 3 1 C SAME ReLU Convolution 3 1 C SAME Linear Add the input and the output Upsampling -C feature maps Nearest-neighbor interpolation, factor 2 Convolution 3 1 C SAME ReLUNormalization Conditional instance normalization after every convolution Optimizer Adam (α = 0.001, β 1 = 0.9, β 2 = 0.999)
Stochastic style transfer with adjustable features. 
{"src": "scitldr-AIC_train", "paper_id": "HJg4E8IFdE"}