This paper presents the first attempt at stereoscopic neural style transfer, which responds to the emerging demand for 3D movies or AR VR. We start with a careful examination of applying existing monocular style transfer methods to left and right views of stereoscopic images separately. This reveals that the original disparity consistency cannot be well preserved in the final stylization results, which causes 3D fatigue to the viewers. To address this issue, we incorporate a new disparity loss into the widely adopted style loss function by enforcing the bidirectional disparity constraint in non-occluded regions. For a practical real-time solution, we propose the first feed-forward network by jointly training a stylization sub-network and a disparity sub-network, and integrate them in a feature level middle domain. Our disparity sub-network is also the first end-to-end network for simultaneous bidirectional disparity and occlusion mask estimation. Finally, our network is effectively extended to stereoscopic videos, by considering both temporal coherence and disparity consistency. We will show that the proposed method clearly outperforms the baseline algorithms both quantitatively and qualitatively.
In the past, re-drawing an image in a particular style required a well-trained artist to do lots of time-consuming manual work. This motivated the development of a plenty of Non-photorealistic Rendering (NPR) algorithms to make the process automatic, such that everyone can be an artist. However, they are usually confined to the specific artistic styles ( , oil paintings and sketches). Gatys @cite_8 were the first to study how to use CNNs to reproduce famous styles on natural images. They leverage CNNs to characterize the visual content and style, and then recombine both for the transferring styles, which is general to various artistic styles.
@cite_8
2615562997 2109253138 2397563792 2624770008
Humans are capable of perceiving a natural scene at a glance and painters, through their representations, push this capacity to the limit by abstraction. Yet we still do it... What about machines? In this paper we address the problem of recognizing the theme (scene type) in digitized paintings. The approach is based on the Convolutional Neural Network framework and the chosen architecture is the recent Residual Network. In the first level of evaluation, we determine the recognition rate of a CNN given a database of 80,000 annotated digitized paintings. In the second level we evaluate the impact of extending the training database with photographs directly and via two domain adaptation functions and thus we are able to assess the abstraction level that CNN is capable to achieve. This paper surveys the field of nonphotorealistic rendering (NPR), focusing on techniques for transforming 2D input (images and video) into artistically stylized renderings. We first present a taxonomy of the 2D NPR algorithms developed over the past two decades, structured according to the design characteristics and behavior of each technique. We then describe a chronology of development from the semiautomatic paint systems of the early nineties, through to the automated painterly rendering systems of the late nineties driven by image gradient analysis. Two complementary trends in the NPR literature are then addressed, with reference to our taxonomy. First, the fusion of higher level computer vision and NPR, illustrating the trends toward scene analysis to drive artistic abstraction and diversity of style. Second, the evolution of local processing approaches toward edge-aware filtering for real-time stylization of images and video. The survey then concludes with a discussion of open challenges for 2D NPR identified in recent NPR symposia, including topics such as user and aesthetic evaluation. We consider the problem of recognizing objects in collections of art works, in view of automatically labeling, searching and organizing databases of art works. To avoid manually labelling objects, we introduce a framework for transferring a convolutional neural network (CNN), trained on available large collections of labelled natural images, to the context of drawings. We retrain both the top and the bottom layer of the network, responsible for the high-level classiication output and the low-level features detection respectively, by transforming natural images into drawings. We apply this procedure to the drawings in the Jan Brueghel Wiki, and show the transferred CNN learns a discriminative metric on drawings and achieves good recognition accuracy. We also discuss why standard descriptor-based methods is problematic in the context of drawings. Recently, with the revolutionary neural style transferring methods, creditable paintings can be synthesized automatically from content images and style images. However, when it comes to the task of applying a painting's style to an anime sketch, these methods will just randomly colorize sketch lines as outputs and fail in the main task: specific style tranfer. In this paper, we integrated residual U-net to apply the style to the gray-scale sketch with auxiliary classifier generative adversarial network (AC-GAN). The whole process is automatic and fast, and the results are creditable in the quality of art style as well as colorization.