This paper presents the first attempt at stereoscopic neural style transfer, which responds to the emerging demand for 3D movies or AR VR. We start with a careful examination of applying existing monocular style transfer methods to left and right views of stereoscopic images separately. This reveals that the original disparity consistency cannot be well preserved in the final stylization results, which causes 3D fatigue to the viewers. To address this issue, we incorporate a new disparity loss into the widely adopted style loss function by enforcing the bidirectional disparity constraint in non-occluded regions. For a practical real-time solution, we propose the first feed-forward network by jointly training a stylization sub-network and a disparity sub-network, and integrate them in a feature level middle domain. Our disparity sub-network is also the first end-to-end network for simultaneous bidirectional disparity and occlusion mask estimation. Finally, our network is effectively extended to stereoscopic videos, by considering both temporal coherence and disparity consistency. We will show that the proposed method clearly outperforms the baseline algorithms both quantitatively and qualitatively.
With the increasing popularity and great business potential of 3D movies or AR VR techniques, stereoscopic image video processing techniques have drawn much attention. Some interesting stereoscopic topics include image inpainting @cite_35 , object copy and paste @cite_43 , image retargeting @cite_23 @cite_36 , image warping @cite_45 and video stabilization @cite_20 . In this work, we introduce a new topic which turns stereoscopic images or videos into synthetic artworks.
@cite_35 @cite_36 @cite_43 @cite_45 @cite_23 @cite_20
2155830983 2129850474 1970835671 1996698198 2128529780 2514361303
We present a novel algorithm for simultaneous color and depth inpainting. The algorithm takes stereo images and estimated disparity maps as input and fills in missing color and depth information introduced by occlusions or object removal. We first complete the disparities for the occlusion regions using a segmentation-based approach. The completed disparities can be used to facilitate the user in labeling objects to be removed. Since part of the removed regions in one image is visible in the other, we mutually complete the two images through 3D warping. Finally, we complete the remaining unknown regions using a depth-assisted texture synthesis technique, which simultaneously fills in both color and depth. We demonstrate the effectiveness of the proposed algorithm on several challenging data sets. Image retargeting algorithms attempt to adapt the image content to the screen without distorting the important objects in the scene. Existing methods address retargeting of a single image. In this paper we propose a novel method for retargeting a pair of stereo images. Naively retargeting each image independently will distort the geometric structure and make it impossible to perceive the 3D structure of the scene. We show how to extend a single image seam carving to work on a pair of images. Our method minimizes the visual distortion in each of the images as well as the depth distortion. A key property of the proposed method is that it takes into account the visibility relations between pixels in the image pair (occluded and occluding pixels). As a result, our method guarantees, as we formally prove, that the retargeted pair is geometrically consistent with a feasible 3D scene, similar to the original one. Hence, the retargeted stereo pair can be viewed on a stereoscopic display or processed by any computer vision algorithm. We demonstrate our method on a number of challenging indoor and outdoor stereo images. With the increase in popularity of stereoscopic 3D imagery for film, TV, and interactive entertainment, an urgent need for editing tools to support stereo content creation has become apparent. In this paper we present an end-to-end system for object copy & paste in a stereoscopic setting to address this need. There is no straightforward extension of 2D copy & paste to support the addition of the third dimension as we show in this paper. For stereoscopic copy & paste we need to handle depth, and our core objective is to obtain a convincing 3D viewing experience. As one of the main contributions of our system, we introduce a stereo billboard method for stereoscopic rendering of the copied selection. Our approach preserves the stereo volume and is robust to the inevitable inaccuracies of the depth maps computed from a stereo pair of images. Our system also includes an interactive stereoscopic segmentation tool to achieve high quality object selection. Hence, we focus on intuitive and minimal user interaction, and our editing operations perform within interactive rates to provide immediate feedback. Warping is one of the basic image processing techniques. Directly applying existing monocular image warping techniques to stereoscopic images is problematic as it often introduces vertical disparities and damages the original disparity distribution. In this paper, we show that these problems can be solved by appropriately warping both the disparity map and the two images of a stereoscopic image. We accordingly develop a technique for extending existing image warping algorithms to stereoscopic images. This technique divides stereoscopic image warping into three steps. Our method first applies the user-specified warping to one of the two images. Our method then computes the target disparity map according to the user specified warping. The target disparity map is optimized to preserve the perceived 3D shape of image content after image warping. Our method finally warps the other image using a spatially-varying warping method guided by the target disparity map. Our experiments show that our technique enables existing warping methods to be effectively applied to stereoscopic images, ranging from parametric global warping to non-parametric spatially-varying warping. We propose a content-aware stereoscopic image display adaptation method which simultaneously resizes a binocular image to the target resolution and adapts its depth to the comfort zone of the display while preserving the perceived shapes of prominent objects. This method does not require depth information or dense correspondences. Given the specification of the target display and a sparse set of correspondences, our method efficiently deforms the input stereoscopic images for display adaptation by solving a least-squares energy minimization problem. This can be used to adjust stereoscopic images to fit displays with different real estates, aspect ratios and comfort zones. In addition, with slight modifications to the energy function, our method allows users to interactively adjust the sizes, locations and depths of the selected objects, giving users aesthetic control for depth perception. User studies show that the method is effective at editing depth and reducing occurrences of diplopia and distortions. This paper presents a method to stabilize shaky stereoscopic videos captured by hand-held devices. Directly applying traditional monocular video stabilization techniques to two views independently is problematic as it often brings undesirable vertical disparities and produces inaccurate horizontal disparities, which violate original stereoscopic disparity constraints, leading to erroneous depth perception. In this paper, we show that monocular video stabilization methods, such as the bundled camera paths stabilization, can be extended for stereoscopic videos by taking additional disparity constraints during the stabilization. In particular, we first estimate disparities between two views. Then, we compute camera motions as meshes of bundled paths for each view. Next, we smooth paths of two views separately and iteratively. During each iteration, we adjust the meshes of one view by our proposed ‘Joint Disparity and Stability mesh Warp (JDSW)’. The final result is generated after several iterations of paths smoothing and meshes adjusting, in which temporal stability and correct depth perception are achieved simultaneously. We evaluate our method by various challenging stereoscopic videos with different camera motions and scene types. The experiments demonstrate the effectiveness of our method.