 Recent advances in deep learning have shown promising results in many low-level vision tasks. However, solving the single-image-based view synthesis is still an open problem. In particular, the generation of new images at parallel camera views given a single input image is of great interest, as it enables 3D visualization of the 2D input scenery. We propose a novel network architecture to perform stereoscopic view synthesis at arbitrary camera positions along the X-axis, or Deep 3D Pan, with "t-shaped" adaptive kernels equipped with globally and locally adaptive dilations. Our proposed network architecture, the monster-net, is devised with a novel t-shaped adaptive kernel with globally and locally adaptive dilation, which can efficiently incorporate global camera shift into and handle local 3D geometries of the target image's pixels for the synthesis of naturally looking 3D panned views when a 2-D input image is given. Extensive experiments were performed on the KITTI, CityScapes and our VXXLXX_STEREO indoors dataset to prove the efficacy of our method. Our monster-net significantly outperforms the state-of-the-art method, SOTA, by a large margin in all metrics of RMSE, PSNR, and SSIM. Our proposed monster-net is capable of reconstructing more reliable image structures in synthesized images with coherent geometry. Moreover, the disparity information that can be extracted from the "t-shaped" kernel is much more reliable than that of the SOTA for the unsupervised monocular depth estimation task, confirming the effectiveness of our method.
 Recent advances in deep learning have pushed forward the state-of-the-art performance for novel view synthesis problems. Novel view synthesis is the task of generating a new view seen from a different camera position, given a single or multiple input images, and finds many applications in robotics, navigation, virtual and augmented reality (VR/AR), cinematography, etc. In particular, the challenging task of generating stereo images given a single input view is of great interest as it enables 3D visualization of the 2D input scene. In addition, the falling price and the increasing availability of the equipment required for VR/AR has fueled the demand for stereoscopic contents. The previous works, such as the Deep3D (Xie et al., 2016) , have addressed the right-view generation problem in a fully supervised fashion when the input is the left-view to which the output is the synthetic right-view at a fixed camera shift. In contrast, our proposed Deep 3D Pan pipeline enables the generation of new views at arbitrary camera positions along the horizontal X-axis of an input image with far better quality by utilizing adaptive "t-shaped" convolutions with globally and locally adaptive dilations, which takes into account the camera shift amount and the local 3D geometries of the target pixels. Panning at arbitrary camera positions allows our proposed model to adjust the baseline (distance between cameras) for different levels of 3D sensation. Additionally, arbitrary panning unlocks the possibility to adjust for different inter-pupillary distances of various persons. Figure 1 shows some generated left and right view images for a given single image input by our proposed Deep 3D Pan pipeline, which we call it the "monster-net" (monocular to stereo network). In this paper, we define "pan" in the context of 3D modeling, implying that camera movement is in parallel to the center view plane. In the following sections, we review the related works to stereoscopic view synthesis and discuss the differences with our proposed method, followed by the formulation of our Deep 3d Pan pipeline and finally, we present outstanding results on various challenging stereo datasets, showing superior performance against the previous state-of-the-art methods. We presented an adaptive "t-shaped" kernel equipped with globally and locally adaptive dilations for the Deep 3D Pan problem, defined as the task of arbitrarily shifting the camera position along the X-axis for stereoscopic view synthesis. Our proposed monster-net showed superior performance to the SOTA for right-view generation on the KITTI and the CityScapes datasets. Our monsternet also showed very good generalization capabilities with 3dB gain in PSNR against the Deep3D baseline. In addition, our method presents no-discontinuities, consistent geometries, good contrast, and naturally looking left or right synthetic panned images. Our monster-net can be extended for image registration, monocular video to stereo video, and generation of novel views at any camera translation by just allowing pixel-wise rotation of our "t-shaped" kernel.
Novel architecture for stereoscopic view synthesis at arbitrary camera shifts utilizing adaptive t-shaped kernels with adaptive dilations.
{"src": "scitldr-AIC_train", "paper_id": "B1gF56VYPH"}