1. A method comprising: receiving a plurality of input images of a scene taken from different vantage points; processing the plurality of input images to generate a sparse reconstruction representation of the scene, the sparse reconstruction representation including a sparse point cloud specifying locations of a plurality of points that correspond to three-dimensional locations of surfaces of objects in the scene; based in part on the sparse reconstruction representation, processing the plurality of input images to generate respective dense reconstruction representations of each of the plurality of input images, the respective dense reconstruction representations each including respective depth images for the plurality of input images, the depth images including color and depth information; projecting front surfaces of the depth images using a forward depth test to generate a plurality of front-warped images; projecting back surfaces of the depth images using an inverted depth test to generate a plurality of back-warped images; stitching the front-warped images and the back-warped images to generate a two-layer panorama having a front surface panorama and a back surface panorama; and fusing the front surface panorama and the back surface panorama in the two-layer panorama to generate the three-dimensional image comprising a multi-layered geometric mesh suitable for rendering the scene in the three-dimensional space.