Out of Sight, Out of Mind: A Source-View-Wise Feature Aggregation for Multi-View Image-Based Rendering
To estimate the volume density and color of a 3D point in the multi-view
image-based rendering, a common approach is to inspect the consensus existence
among the given source image features, which is one of the informative cues for
the estimation procedure. To this end, most of the previous methods utilize
equally-weighted aggregation features. However, this could make it hard to
check the consensus existence when some outliers, which frequently occur by
occlusions, are included in the source image feature set. In this paper, we
propose a novel source-view-wise feature aggregation method, which facilitates
us to find out the consensus in a robust way by leveraging local structures in
the feature set. We first calculate the source-view-wise distance distribution
for each source feature for the proposed aggregation. After that, the distance
distribution is converted to several similarity distributions with the proposed
learnable similarity mapping functions. Finally, for each element in the
feature set, the aggregation features are extracted by calculating the weighted
means and variances, where the weights are derived from the similarity
distributions. In experiments, we validate the proposed method on various
benchmark datasets, including synthetic and real image scenes. The experimental
results demonstrate that incorporating the proposed features improves the
performance by a large margin, resulting in the state-of-the-art performance.