Interest point detection and local feature description are fundamental steps in many computer vision applications. Classical methods for these tasks are based on a detect-then-describe paradigm where separate handcrafted methods are used to first identify repeatable keypoints and then represent them with a local descriptor. Neural networks trained with metric learning losses have recently caught up with these techniques, focusing on learning repeatable saliency maps for keypoint detection and learning descriptors at the detected keypoint locations. In this work, we argue that salient regions are not necessarily discriminative, and therefore can harm the performance of the description. Furthermore, we claim that descriptors should be learned only in regions for which matching can be performed with high confidence. We thus propose to jointly learn keypoint detection and description together with a predictor of the local descriptor discriminativeness. This allows us to avoid ambiguous areas and leads to reliable keypoint detections and descriptions. Our detection-and-description approach, trained with self-supervision, can simultaneously output sparse, repeatable and reliable keypoints that outperforms state-of-the-art detectors and descriptors on the HPatches dataset. It also establishes a record on the recently released Aachen Day-Night localization dataset.
More recently, D2-Net @cite_41 uses a single CNN for joint detection and description that share all weights; the detection being based on local maxima across the channels and the spatial dimensions of the feature maps. Instead of arbitrarily defining keypoints as local maxima in the descriptor space, our approach explicitly estimates the keypoint reliability and repeatability. Finally, Ono al @cite_10 train a network from pairs of matching images with a complicated asymmetric gradient backpropagation scheme for the detection and a triplet loss for the local descriptor. Compared to these works, for the first time we jointly train a sparse keypoint detector with a deep descriptor enhanced with a reliability confidence value such that ambiguous areas are avoided.
@cite_41 @cite_10
2944011193 2963537932
In this work we address the problem of finding reliable pixel-level correspondences under difficult imaging conditions. We propose an approach where a single convolutional neural network plays a dual role: It is simultaneously a dense feature descriptor and a feature detector. By postponing the detection to a later stage, the obtained keypoints are more stable than their traditional counterparts based on early detection of low-level structures. We show that this model can be trained using pixel correspondences extracted from readily available large-scale SfM reconstructions, without any further annotations. The proposed method obtains state-of-the-art performance on both the difficult Aachen Day-Night localization dataset and the InLoc indoor localization benchmark, as well as competitive performance on other benchmarks for image matching and 3D reconstruction. We present a novel strategy to learn a local feature pipeline from collections of images with deep networks, without the need for human supervision. To do so we frame the learning problem with a two-branch network. We posit that training both branches with a standard Siamese architecture is not feasible, as solving the correspondence problem jointly with feature learning is too challenging, and does not converge well enough to train from scratch. Instead, we propose to break differentiability on one branch and use ground-truth geometry to remove the burden of solving the correspondence problem while training, while keeping the other fully-differentiable. In order to train this setup with gradient-based methods, we optimize for the differentiable branch while using the parameters from the previous training step for the other, and demonstrate that both converge to a single, optimal solution. Our method can be trained with only the relative camera pose and depth information---furthermore, we show that this ground truth does not need to be perfect and can be easily obtained with off-the-shelf Structure-from-Motion solutions. Our models outperform the state of the art on sparse feature matching on both indoor and outdoor datasets, while running at 60+ fps for QVGA images.