Iterative Depth Estimation.    A few methods have adopted an iterative depth estimation paradigm. R-MVSNet [1]} iteratively regularizes each slice of the cost volume with GRU along the depth dimension. Unlike those cost volume based approaches before mentioned, Point-MVSNet [2]} directly processes the target scene as point clouds. It first generates a coarse depth map, converts it into a point cloud and refines the point cloud iteratively to reduce the residual between the currently estimated depth and the ground truth depth. IterMVS [3]} proposes to encode a pixel-wise probability distribution of depth in the hidden state of a GRU-based estimator. Each iteration, the multi-scale matching information is injected into the GRU to predict the depth and confidence maps to facilitate the following 3D reconstruction. The depth maps are predicted via a combined classification and regression through the probability distribution. It uses arg-min to finish the "classification" function, which is not differentiable and must be detached first before the its following regression operation. Unlike IterMVS, our method learns to recurrently index the cost volume to directly find the best depth candidates in an end-to-end, differentiable way.
