Computer Science
Computer Science-2021.gz:143318
Computer Science
534ff47a5bd665ae38398053a90f7295a0e0b30b
3DVNet: Multi-View Depth Prediction and Volumetric Refinement

We present 3DVNet, a novel multi-view stereo (MVS) depth-prediction method that combines the advantages of previous depth-based and volumetric MVS approaches. Our key idea is the use of a 3D scene-modeling network that iteratively updates a set of coarse depth predictions, resulting in highly accurate predictions which agree on the underlying scene geometry. Unlike existing depth-prediction techniques, our method uses a volumetric 3D convolutional neural network (CNN) that operates in world space on all depth maps jointly. The network can therefore learn meaningful scene-level priors. Furthermore, unlike existing volumetric MVS techniques, our 3D CNN operates on a feature-augmented point cloud, allowing for effective aggregation of multi-view information and flexible iterative refinement of depth maps. Experimental results show our method exceeds state-of-the-art accuracy in both depth prediction and 3D reconstruction metrics on the ScanNet dataset, as well as a selection of scenes from the TUM-RGBD and ICL-NUIM datasets. This shows that our method is both effective and generalizes to new settings.


Introduction
Multi-view stereo (MVS) is a central problem in computer vision with applications from augmented reality to autonomous navigation. In MVS, the goal is to reconstruct a scene using only posed RGB images as input. This reconstruction can take many forms, from voxelized occupancy or truncated signed distance fields (TSDFs), to per-frame depth prediction, the focus of this paper. In recent years, MVS methods based on deep learning [2,6,11,12,17,18,22,24,26,29,30,31,32] have surpassed traditional MVS methods [9,21] on numerous benchmark datasets [5,13,15]. In this work, we consider these methods as falling into two categories, depth estimation and volumetric reconstruction, each with advantages and disadvantages.
The most recent learning methods in depth estimation https://github.com/alexrich021/3dvnet Figure 1: Volumetric methods lack local detail while depthbased methods lack global coherence. Our method cyclically predicts depth, back-projects into 3D space, volumetrically models geometry, and updates all depth predictions to match, resulting in local detail and global coherence.
use deep features to perform dense multi-view matching robust to large environmental lighting changes and textureless or specular surfaces, among other things. These methods take advantage of well researched multi-view aggregation techniques and the flexibility of depth as an output modality. They formulate explicit multi-view matching costs and include iterative refinement layers in which a network predicts a small depth offset between an initial prediction and the ground truth depth map [2,32]. While these techniques have been successful for depth prediction, most are constrained to making independent, per-frame predictions. This results in predictions that do not agree on the underlying 3D geometry of the scene. Those that do make joint predictions across multiple frames use either regularization constraints [11] or recurrent neural networks (RNNs) [6] to encourage frames close in pose space to make similar predictions. However, these methods do not directly operate on a unified 3D scene representation, and their resulting reconstructions lack global coherence (see Fig. 1). Meanwhile, volumetric techniques operate directly on a unified 3D scene representation by back-projecting and aggregating 2D features into a 3D voxel grid and using a 3D convolutional neural network (CNN) to regress a voxelized parameter, often a TSDF. These methods benefit from the use of 3D CNNs and naturally produce highly coherent 3D reconstructions and accurate depth predictions. However, they do not explicitly formulate a multi-view matching cost like depth-based methods, generally averaging deep features from different views to populate the 3D voxel grid. This results in overly-smooth output meshes (see Fig. 1).
In this paper, we propose 3DVNet, an end-to-end differentiable method for learned multi-view depth prediction that leverages the advantages of both volumetric scene modeling and depth-based multi-view matching and refinement.
The key idea behind our method is the use of a 3D scenemodeling network which outputs a multi-scale volumetric encoding of the scene. This encoding is used with a modified PointFlow algorithm [2] to iteratively update a set of initial coarse depth predictions, resulting in predictions that agree on the underlying scene geometry.
Our 3D network operates on all depth predictions at once, and extracts meaningful, scene-level priors similar to volumetric MVS methods. However, the 3D network operates on features aggregated using depth-based multi-view matching and can be used iteratively to update depth maps. In this way, we combine the advantages of the two separate classes of techniques. Because of this, 3DVNet exceeds state-of-the-art results on ScanNet [5] in nearly all depth map prediction and 3D reconstruction metrics when compared with the current best depth and volumetric baselines. Furthermore, we show our method generalizes to other real and synthetic datasets [10,23], again exceeding the best results on nearly all metrics. Our contributions are as follows: 1. We present a 3D scene-modeling network which outputs a volumetric scene encoding, and show its effectiveness for iterative depth residual prediction.
2. We modify PointFlow [2], an existing method for depth map residual predictions, to use our volumetric scene encoding.
3. We design 3DVNet, a full MVS pipeline, using our 3D scene-modeling network and PointFlow refinement.

Related Works
We cover MVS methods using deep learning, categorizing them as either depth-prediction methods or volumetric methods. Our method falls into the first category, but is very much inspired by volumetric techniques.
Depth-Prediction MVS Methods: With some notable exceptions [22,28], nearly all depth-prediction methods follow a similar paradigm: (1) they construct a plane sweep cost volume on a reference image's camera frustum, (2) they fill the volume with deep features using a cost function that operates on source and reference image features, (3) they use a network to predict depth from this cost volume. Most methods differ in their cost metric used to construct the volume. Many cost metrics exist, including per-channel variance of deep features [29,30], learned aggregation using a network [17,31], concatenation of deep features [12], the dot product of deep features [6], and absolute intensity difference of raw image RGB values [11,26]. We find perchannel variance [29] to be the most commonly used cost metric, and adopt it in our system.
The choice of cost aggregation method results in either a vectorized matching cost and thus a 4D cost volume [2,12,17,29,30,31,32] or a scalar matching cost and thus a 3D cost volume [6,11,26]. Methods with 4D cost volumes generally require 3D networks for processing, while 3D cost volumes can be processed with a 2D U-Netstyle [20] encoder-decoder architecture. Some methods operate on the deep features at the bottleneck of this U-Net to make joint depth predictions for all N frames or a subset of frames in a given scene. This is similar to our proposed method, and we highlight the differences.
GPMVS [11] uses a Gaussian Process (GP) constraint conditioned on pose distance to regularize these deep features. This GP constraint only operates on deep features and assumes Gaussian priors. In contrast, we directly learn priors from predicted depth maps and explicitly predict depth residuals to modify depth maps to match. DV-MVS [6] introduces an RNN to propagate information from the deep features in frame t − 1 to frame t given an ordered sequence of frames. While they do propagate this information in a geometrically plausible way, the RNN operates only on deep features similar to GPMVS. Furthermore, the RNN never considers all frames jointly like our method.
Similar to our method, some networks iteratively predict a residual to refine an initial depth prediction [2,32]. We specifically highlight Point-MVSNet [2], which introduces PointFlow, a point cloud learning method for residual prediction. Our method is very much inspired by this work. We briefly describe the differences.
In their work, they operate on a point cloud backprojected from a single depth map and augmented with additional points. Features are extracted from this point cloud using point cloud learning techniques and used in their PointFlow module for residual prediction. Crucially, these features do not come from a unified 3D representation of the scene. Thus the residual prediction is only conditioned on information local to the individual depth prediction and not global scene information. In contrast, our variation of PointFlow uses our volumetric scene model to condition residual prediction on information from all depth maps. For an in depth discussion of differences, see Sec. 3.2.
Volumetric MVS Methods: In volumetric MVS, the goal is to directly regress a global volumetric representation of the scene, generally a TSDF volume. We highlight two methods which inspired our work. Atlas [18] backprojects rays of deep features extracted from images into a global voxel grid, pools features from multiple views using a running average, then directly regress a TSDF in a coarseto-fine fashion using a 3D U-Net. NeuralRecon [24] improves on the memory consumption and run-time of Atlas by reconstructing local fragments using the most recent keyframes, then fusing the local fragments to a global volume using an RNN. The reconstructions these methods produce are pleasing. However, both construct feature volumes using averaging in a single forward pass, which we believe is non-optimal. In contrast, our depth-based method allows us to construct a feature volume using multi-view matching features and perform iterative refinement.

Methods
Our method takes as input N images, denoted {I n }, n = 1, . . . , N with corresponding known extrinsic and intrinsic camera parameters. Our goal is to predict N depth maps {D n } corresponding to the N images. As a preprocessing step, we define for every image I n a set of M indices {s 1 , . . . , s M } pointing to which images to use as source images for depth prediction, and append the reference index to form the set S n = {n, s 1 , . . . , s M }.
Our pipeline is as follows. First, a small depth-prediction network is used to independently predict initial coarse depth maps {D 0 n } for every frame {I n } using extracted image features {F n } (Sec. 3.3). Second, we back-project our N initial depth maps to form a joint point cloud X ⊂ R 3 (Sec 3.1). Because each point p ∈ X is associated with one depth map D 0 n that has associated feature maps {F s : s ∈ S n }, we can augment it with a multi-view matching feature aggregated from those feature maps. Third, our 3D scene-modeling network takes as input this featurerich point cloud and outputs a multi-scale scene encoding V 1 , V 2 , V 3 (Sec. 3.1). Fourth, we update each depth map to match this scene encoding using a modified PointFlow algorithm, resulting in highly coherent depth maps and thus highly coherent reconstructions (Sec. 3.2). Steps 2-4 can be run in a nested for-loop, with steps 2 and 3 run in the outer loop to generate updated scene models with the current depth maps and step 4 run in the inner loop to refine depth maps with the current scene model. We denote the updated depth map after l o outer loop iterations of scene modeling and l i inner loop iterations of updating as D

3D Scene Modeling
A visualization of our 3D scene modeling method is given in the upper half of Fig. 2. As stated previously, our 3D scene-modeling network operates on a feature rich point cloud back-projected from {D 0 n } or subsequent updated depth maps. To process this point cloud, we adopt a voxelize-then-extract approach. We first generate a sparse 3D grid of voxels, culling voxels that do not contain depth points. To avoid losing granular information of the point cloud, we generate a deep feature for each voxel using a pervoxel PointNet [1]. The PointNet inputs are the features of each depth point in the voxel as well as the 3D offset of that point to the voxel center. Finally, we run a 3D U-Net [20] on the resulting voxelized feature volume and extract intermediate outputs at multiple resolutions. By nature of construction, this U-Net learns meaningful, scene-level priors. The result is a multi-scale, volumetric scene encoding.
Point Cloud Formation: We form our point cloud X ⊂ R 3 by back-projecting all depth pixels in all N depth maps. For our multi-view matching feature associated with each point p ∈ X, we follow existing work [2,29] and use per-channel variance aggregation using the reference and source feature maps associated with each depth pixel. For p ∈ X, given that p belongs to depth map D 0 n , the equation for variance feature σ 2 (p), applied per-channel, is: is the bilinear interpolation of F s to pointp s , and F * (p * ) is the average interpolated feature over all indices s ∈ S n . Intuitively, if p lies on a surface it is more likely to have low variance in most feature channels in σ 2 (p) while if it doesn't lie on a surface the variance will likely be high.
Point Cloud Voxelization: To form our initial feature volume, we regularly sample an initial 3D grid of points C every r = 8 cm within the axis-aligned bounding box of point cloud X and define the voxel associated with each grid point c ∈ C as the 8 cm 3 cube with center c. We denote the set of depth points that fall within a voxel with center c ∈ C as v(c) = {p ∈ X : ||c − p|| ∞ <= r 2 }. We sparsify this grid by discarding c ∈ C if no depth points lie within the associated voxel, denoting this set of grid coordinates asĈ = {c ∈ C : v(c) = ∅}. For c ∈Ĉ, we produce a feature for the associated voxel using PointNet [1] with max pooling. The PointNet feature for each voxel is defined as: where h θ is a learnable multi-layer perceptron (MLP), concat [q, f ] indicates concatenation of the 3D coordinates q with the feature channel of f to form a feature with 3 additional channels, and is the channel-wise max pooling operation. The result of this stage is a sparse feature volume V 0 with features given by Eq. 2 and coordinatesĈ.  Multi-Scale 3D Feature Extraction: In this stage, we use a sparse 3D U-Net to model the underlying scene geometry. We use a basic U-Net architecture with skip connections. Group normalization is used throughout. See supplementary material for a more detailed description of our architecture. Our sparse U-Net takes as input sparse feature volume V 0 . From intermediate outputs of the U-Net, we extract three scales of feature volumes V 1 , V 2 , V 3 with a voxel edge length of 4r = 32 cm, 2r = 16 cm, and r = 8 cm, respectively, describing the scene. In this way, we extract a rich, multi-scale, volumetric encoding of the scene.

PointFlow-Based Refinement
In this stage, we use our multi-scale scene encoding V 1 , V 2 , V 3 from the previous stage in a variation of the PointFlow algorithm proposed by Chen et al. [2]. The goal is to refine our predicted depth maps to match our scene model by predicting a residual for each depth pixel. We briefly review the core components of PointFlow and the intuition behind our proposed change.
In PointFlow, a set of points called hypothesis points are constructed at regular intervals along a depth ray, centered about the depth prediction associated with the given depth ray. The blue and red points in Fig. 3 illustrate this. Features are generated for the hypothesis points. Then, a network processes these features and outputs a probability score for every point indicating confidence the given point is at the correct depth. Finally, the expected offset is calculated using these probabilities and added to the original depth prediction. Our key innovation is the use of our multi-scale scene encoding to generate the hypothesis point features.
In the original PointFlow, hypothesis points are constructed for a single depth map, augmented with features using Eq. 1, and aggregated into a point cloud. Note this point cloud is strictly different from our point cloud as (1) it is produced using a single depth map, and (2) it includes hypothesis points. Features are generated for each point using edge convolutions [27] on the k-Nearest-Neighbor (kNN) graph. Crucially, these edge convolutions never operate on a unified 3D scene representation in the original PointFlow. This prevents the offset predictions from learning global information, which we believe is a critical step for depth residual prediction. Furthermore, because of the required kNN search, this formulation cannot scale to process a joint point cloud from an arbitrary number of depth maps, therefore preventing it from scaling to learn global information.
Inspired by convolutional occupancy networks [19] and IFNets [3], we instead generate hypothesis features by interpolating each scale of our multi-scale scene encoding (see Fig. 3). With this key change, we use powerful scene-level priors in our offset prediction conditioned on all N depth predictions for a given scene. Furthermore, by using the same encoding to update all N depth predictions, we encourage global consistency of predictions. We now describe in detail our variation of the PointFlow method (see Figs. 2 and 3), using notation similar to the original paper.
Hypothesis Point Construction: For a given backprojected depth pixel p from depth map D n , we generate 2h + 1 point hypotheses {p k }: where t is the normalized reference camera direction of D n , and s is the displacement step size. Feature Generation: We generate a multi-scale feature for each hypothesis pointp k using trilinear interpolation to pointp k of our sparse features volumes V 1 , V 2 , V 3 , using 0s where features are not defined: Next, we generate a variance feature σ 2 (p k ) for hypothesis pointp k using Eq. 1. The final feature for a hypothesis point is the channel-wise concatenation of these features: We stack our 2h + 1 point-hypothesis features to form a 2D feature H ∈ R (2h+1)×c , where c is the sum of the dimensions of our variance and scene encoding features. Offset Prediction: We apply a 4 layer 1D CNN followed by a softmax function to predict a probability scalar for each point-wise entry in H. The predicted displacement of point p is then as follows: The updated depth for each depth map is the depth of point p + t∆d p with respect to the camera associated with D n .

Bringing It All Together: 3DVNet
In this section, we describe our full depth-prediction pipeline using our multi-scale volumetric scene modeling and PointFlow-based refinement, which we name 3DVNet (see Fig. 4). Our pipeline consists of (1) initial feature extraction and depth prediction, (2) scene modeling and refinement, and (3) upsampling of our refined depth map to the size of the original image. The scene modeling and refinement is done in a nested for-loop fashion, extracting a scene model in the outer loop and iteratively refining the depth predictions using that scene model in the inner loop. We fix the input image size of 3DVNet to 320 × 256.
2D Feature Extraction: For our 2D feature extraction, we adopt the approach of Düzçeker et al. [6], and use a 32 channel feature pyramid network (FPN) [16] constructed on a MnasNet [25] backbone to extract coarse and fine resolution feature maps of size 80×64 and 160×128 respectively. For every image I n , we denote these F c n and F f n . MVSNet Prediction: For the coarse depth prediction of image I n , we use a small MVSNet [29] using the reference and source coarse feature maps {F c s : s ∈ S n } to predict an initial coarse depth D 0 n . Our cost volume is constructed using traditional plane sweep stereo with L = 96 depth hypotheses sampled uniformly at intervals of 5 cm starting at 50 cm. Similar to Yu and Gao [32], our predicted depth map is spatially sparse compared to feature map F c n . We fix our coarse depth map prediction size to 56 × 56.
Nested For-Loop Refinement: We denote the updated depths after scene-modeling iteration l o and PointFlow iteration l i as {D (lo,li) n }. We use initial depth predictions {D 0 n } and coarse feature maps {F c n } to generate multi-scale scene encoding V 1 , V 2 , V 3 . We then run PointFlow refinement three times with displacement step size s = 5 cm, 5 cm, and 2.5 cm and h = 3 to get updated depths {D In early experiments, we found two iterations at 5 cm to be helpful. We re-generate our scene encoding using updated depths {D We find our depth maps converge at this point.
Coarse-to-Fine Upsampling: In this stage, we upsample each refined depth prediction D (2,3) n to the size of image I n . We find PointFlow refinement does not remove interpolation artifacts, as this generally requires predicting large offsets across depth boundaries. We outline a simple, coarse-to-fine method for upsampling while removing artifacts. See the right section of Fig. 4. At each step, we upsample the current depth prediction using nearestneighbor interpolation to the size of the next-largest feature map and concatenate, using the original image I n in the final step. We then pass the concatenated feature map and depth through a smoothing network. We use a version of the propagation network proposed by Yu and Gao [32]. For every pixel p in depth map D, the smoothed depthD is a weighted sum of D in the 3 × 3 neighborhood about p: where g θ is a 4 layer CNN that takes as input the concatenated feature and depth map and outputs 9 weights for every pixel p, and g θ (p, q) indexes those weights for the pixel p.
A softmax function is applied to the weights for normalization. We apply this coarse-to-fine upsampling to every refined depth map {D

Implementation and Training Details
Libraries: Our model is implemented in PyTorch using PyTorch Lightning [7] and PyTorch Geometric [8]. We use Minkowski Engine [4] as our sparse tensor library. We use Open3D [33] for both visualization and evaluation.
Training Parameters: We train our network on a single NVIDIA RTX 3090 GPU. Our network is trained endto-end with a mini-batch size of 2. Each mini-batch consists of 7 images for depth prediction. For our loss function, we accumulate the average L 1 error between ground truth and predicted depth maps, appropriately downsampling the ground truth depth map to the correct resolution, for all predicted, refined, and upsampled depth map at every stage in our pipeline. Additionally, we employ random geometric scale augmentation with a factor selected between 0.9 to 1.1 and random rotation about the gravitational axis.
We first train with the pre-trained MnasNet backbone frozen using the Adam optimizer [14] with an initial learning rate of 10 −3 which is divided by 10 every 100 epochs (∼1.5k iterations), to convergence (∼1.8k iterations). We unfreeze the MnasNet backbone and finetune the entire network using Adam and an initial learning rate of 10 −4 that is halved every 50 epochs to convergence (∼1.8k iterations).

Datasets, Baselines, Metrics, and Protocols
Datasets: To train and validate our model, we use the ScanNet [5] official training and validation splits. For our main comparison experiment, we use the ScanNet official test set, which consists of 100 test scenes in a variety of indoor settings. To evaluate the generalization ability of our model, we select 10 sequences from TUM-RGBD [23], and 4 sequences from ICL-NUIM [10] for comparison.
Baselines: We compare our method to seven state of the art baselines: Point-MVSNet (PMVS) [2], Fast-MVSNet (FMVS) [32], DeepVideoMVS pair/fusion networks (DVMVS pair/fusion) [6], GPMVS batched [11], Atlas [18], and NeuralRecon [24]. The first five baselines are depth-prediction methods while the last two are volumetric methods. Of these, we consider GPMVS and Atlas the most relevant depth and volumetric methods respectively, as both use information from all frames simultaneously during inference. We use the ScanNet training scenes to fintetune methods not trained on ScanNet [2,11,32]. We report both the finetuned and pretrained results, denoted with and without "FT". To account for range differences between the DTU dataset [13] and ScanNet, we use our model's plane sweep parameters with PMVS and FMVS.
Metrics: We use the 2D and 3D metrics presented by Murez et al. [18] for evaluation. See supplementary for definitions. Amongst these metrics, we consider Abs-rel, Abs-diff, and the first inlier ratio metric δ < 1.25 as the most suitable 2D metrics for measuring depth prediction quality, and F-score as the most suitable 3D metric for measuring 3D reconstruction quality. Following Düzçeker et al. [6], we only consider ground truth depth values greater than 50 cm to account for some methods not being able to predict smaller depth. We note F-score, Precision, and Recall are calculated per-scene and then averaged across all the scenes. This results in a different F-score than when calculating from the averaged Precision and Recall reported.
Protocols: For depth-based methods, we fuse predicted depths using the standard multi-view consistency based point cloud fusion. Based on results on validation sets, we modify the implementation of Galliani et al. [9] to use depth-based multi-view consistency check, rather than a disparity-based check (see Sec. 3.3 of the supplementary materials). For volumetric methods, we use marching cubes to extract a mesh from the predicted TSDF. Following Murez et al. [18], we trim the meshes to remove geometry not observed in the ground truth camera frustums. Additionally, ScanNet ground truth meshes often contain holes in observed regions. We mask out these holes for all baselines to avoid false penalization. All meshes are single layer to match ScanNet ground truth as noted by Sun et al. [24].
We use the DVMVS keyframe selection. For depthbased methods, we use each keyframe as a reference image for depth prediction. We use the 2 previous and 2

Evaluation Results and Discussion
See Tab. 1 for 2D depth and 3D geometry metrics on all datasets. Our method outperforms all baselines by a wide margin on most metrics. Notably, our Abs-rel error on ScanNet is 0.021 less than the DVMVS fusion, the second best method, while the Abs-rel of the third, fourth, and fifth best methods are all within 0.002 of DVMVS fusion. Similarly, Our ScanNet F-score is 0.09 more than GPMVS (FT), the second best method, while the F-score is within 0.001 of GPMVS (FT) for Atlas, the third best method. This demonstrates the significant quantitative increase in both depth and reconstruction metrics of our method. Results on additional datasets show the strong generalization ability of our model.
We include qualitative results on ScanNet. See Figs. 5 and 6. See Sec. 4 of the supplementary materials for additional qualitative results. Our depth maps are visually pleasing, with clearly defined edges. They are comparable in quality to those of GPMVS and DVMVS fusion while being quantitatively more accurate. Our reconstructions are coherent like volumetric methods, without the noise present in other depth-based reconstructions, which we believe is a result of our volumetric scene encoding and refinement.
We do note one benefit of Atlas is its ability to fill large unobserved holes. Though not reflected in the metrics, this leads to qualitatively more complete reconstructions. Our system relies on depth maps and thus cannot do this as designed. However, as a result of averaging across image features, Atlas produces meshes that are overly smooth and lack detail. In contrast, our reconstructions contain sharper, better defined features than purely volumetric methods. Finally, we note our system cannot naturally be run in an online fashion, requiring availability of all frames prior to use.

Ablation and Additional Studies
Does Iterative Refinement Help? We study the effect of each inner and outer loop iteration of our depth refinement. See Tab. 2. We exceed state-of-the-art metrics after 2 iterations. 3 additional iterations add continued improvement, confirming the effectiveness of iterative refinement. By 5 iterations, our metrics have converged, with depth sta-     bilizing and F-score decreasing slightly. Interestingly, the final iteration appears slightly detrimental. Does Multi-Scale Scene Modeling Help? To test this, we (1) completely remove our multi-scale scene encoding from the PointFlow refinement, and (2) only use the coarsest scale V 3 , respectively denoted "no 3D" and "single scale" in Tab. 3. Without any scene-level information, our refinement breaks down, indicating the scene modeling is essential. The single scale model does slightly worse, confirming the effectiveness of our multi-scale encoding.
Do Multi-View Matching Features Help? We use a per-channel average instead of variance aggregation for each point in our feature-rich point cloud, denoted "avg feats" in Tab. 3. Most metrics, notably the F-score, suffer. This supports our hypothesis that multi-view matching is more beneficial for reconstruction than averaging.
For additional studies, see the supplementary material.

Conclusion
We present 3DVNet, which uses the advantages of both depth-based and volumetric MVS. We perform experiments with 3DVNet to show depth-based iterative refinement and multi-view matching combined with volumetric scene modeling greatly improves both depth-prediction and reconstruction metrics. We believe our 3D scene-modeling network bridges an important gap between depth prediction, image feature aggregation, and volumetric scene modeling and has applications far beyond depth-residual prediction. In future work, we will explore its use for segmentation, normal estimation, and direct TSDF prediction.