Most of learning-based MVS methods [1]}, [2]}, [3]}, [4]}, [5]}, [6]} rely on traditional plane-sweeping [7]}, [8]} to generate a cost volume by comparing the CNN features of reference image and source images for matching at several depth hypotheses, and then apply 2D or 3D convolutional encoder-decoder architectures to aggregate and regularize the cost volume. Based on that, they will either apply 2D convolution (by reducing the depth channels via summation) to directly regress a depth map [1]}, or apply soft-argmin [10]} to predict an expected depth from the distribution of the cost volume. The 2D CNN methods, use multi-level features as the skip connections to help decode the cost volume for depth regression. Even though the skip-connection from features improves the depth maps, they weaken the role of cost volume and the geometry knowledge embedded therein to some extend, and hence suffer from degraded generalization when testing on unseen domains. The 3D CNN methods use the soft-argmin to regress the depth map as the expectation from the cost volume distribution, and hence cannot handle well a flat or multi-modal distribution, e.g., caused by textureless, repeated or occluded regions, etc. Therefore, when this situation happens, the soft-argmin cannot well “classify” to target the best candidate, but instead gives an “average” prediction, by aggregating the undetected best candidate, the worst ones and the ones in the middle.
