Residual Pose Net.    An accurate cost volume benefits the GRU-based iterative optimization. The quality of the generated cost volume \(C_0\)  is not only determined by the matching features (\(f_0^a\)  and \(f^{ \mathcal {S}}\) ) (for which we have proposed an asymmetric employment of the transformer layer), but also by the homography warping as in Eq. (REF ). But the camera poses are, in practice, usually obtained by Visual SLAM algorithms [1]}, [2]}, [3]}, and inevitably contains noise. Therefore, we propose a residual pose net to rectify the camera poses for accurately backward warping the reference features to match the corresponding features in source views. We use an image-net pretrained ResNet18 [4]} backbone as in [5]}, [6]} to encode the reference image and the warped source images. Specifically, given the current estimated depth map \(D_t\)  at iteration \(t\) , and the ground truth depth \(D_{gt}\) , we warp a source image \(I_i\)  into the reference view through the homography defined in Eq. (REF ) with (noisy) ground truth camera poses \(\Theta \)  and the \(D_t\)  or \(D_{gt}\) . We randomly select \(D_t\)  or \(D_{gt}\)  with a probability prob(\(D_t\) )=0.6 during network training, but always use the predicted depth \(D_t\)  during network inference. The input to the pose net is the concatenated \(I_0\)  and the warped \(\tilde{I}_i\) , and the output is an axis-angle representation, which is further converted to a residual rotation matrix \(\Delta \theta _i\) , for an updated one \(\theta ^\prime _i\) = \(\Delta \theta _i \cdot \theta _i\) . This way, we predict the residual poses \(\Delta \Theta = \lbrace  \Delta \theta _i \rbrace _{i=1}^{N-1}\)  for each of the source and reference pair, and do the rectification as \(\Theta ^\prime = \Delta \Theta \cdot \Theta \) . We leverage the updated poses \(\Theta ^\prime \)  to calculate a more accurate cost volume \(C_1\)  using Eq. (REF ), followed by the rest of iterations of GRU.
