In this paper, we propose the 3DFeat-Net which learns both 3D feature detector and descriptor for point cloud matching using weak supervision. Unlike many existing works, we do not require manual annotation of matching point clusters. Instead, we leverage on alignment and attention mechanisms to learn feature correspondences from GPS INS tagged 3D point clouds without explicitly specifying them. We create training and benchmark outdoor Lidar datasets, and experiments show that 3DFeat-Net obtains state-of-the-art performance on these gravity-aligned datasets.
The increasing success and popularity of deep learning has recently inspired many learned 3D features. 3DMatch @cite_28 uses a 3D convolutional network to learn local descriptors for indoor RGB-D matching by training on matching and non-matching pairs. PPFNet @cite_14 operates on raw points, incorporating point pair features and global context to improve the descriptor representation. Other works such as CGF @cite_0 and LORAX @cite_34 utilize deep learning to reduce the dimension of their handcrafted descriptors. Despite the good performance of these works, none of them learns to detect keypoints. The descriptors are either computed on all or random sampled points. On the other hand, @cite_1 learns to detect keypoints that give good matching performance with handcrafted SHOT @cite_17 descriptors. This provides the intuition for our work, i.e. a good keypoint is one that gives a good descriptor performance.
@cite_14 @cite_28 @cite_1 @cite_0 @cite_34 @cite_17
2566265240 1972485825 2293670510 2059697919
Matching local geometric features on real-world depth images is a challenging task due to the noisy, low-resolution, and incomplete nature of 3D scan data. These difficulties limit the performance of current state-of-art methods, which are typically based on histograms over geometric properties. In this paper, we present 3DMatch, a data-driven model that learns a local volumetric patch descriptor for establishing correspondences between partial 3D data. To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions. Experiments show that our descriptor is not only able to match local geometry in new scenes for reconstruction, but also generalize to different tasks and spatial scales (e.g. instance-level object model alignment for the Amazon Picking Challenge, and mesh surface correspondence). Results show that 3DMatch consistently outperforms other state-of-the-art approaches by a significant margin. Code, data, benchmarks, and pre-trained models are available online at http: 3dmatch.cs.princeton.edu. AbstractThis paper presents alocal3D descriptor forsurface matching dubbed SHOT.Our proposal stems from a taxonomy of existing methods which highlightstwo major approaches, referred to as Signatures and Histograms, inher-ently emphasizing descriptiveness and robustness respectively. We formulatea comprehensive proposal which encompasses a repeatable local referenceframe as well as a 3D descriptor, the latter featuring an hybrid structurebetween Signatures and Histograms so as to aim at a more favorable bal-ance between descriptive power and robustness. A quite peculiar trait of ourmethod concerns seamless integration of multiple cues within the descrip-tor to improve distinctiveness, which is particularly relevant nowadays dueto the increasing availability of aﬀordable RGB-D sensors which can gatherboth depth and color information. A thorough experimental evaluation basedon datasets acquired with diﬀerent types of sensors, including a novel RGB-D dataset, vouches that SHOT outperforms state-of-the-art local descriptorsin experiments addressing descriptor matching for object recognition, 3Dreconstruction and shape retrieval.Keywords: Surface Matching, 3D Descriptors, Object Recognition, 3DReconstruction1. IntroductionAutomatic recognition of shapes in 3D data, also referred to as surfacematching, attracts growing interest in the research community, with appli-cation to areas such as shape retrieval, 3D reconstruction, object recogni-tion categorization, manipulation and grasping, robot localization and nav-igation. A key enabling factor for the development of this technology is In this paper we propose a new approach for learning local descriptors for matching image patches. It has recently been demonstrated that descriptors based on convolutional neural networks (CNN) can significantly improve the matching performance. Unfortunately their computational complexity is prohibitive for any practical application. We address this problem and propose a CNN based descriptor with improved matching performance, significantly reduced training and execution time, as well as low dimensionality. We propose to train the network with triplets of patches that include a positive and negative pairs. To that end we introduce a new loss function that exploits the relations within the triplets. We compare our approach to recently introduced MatchNet and DeepCompare and demonstrate the advantages of our descriptor in terms of performance, memory footprint and speed i.e. when run in GPU, the extraction time of our 128 dimensional feature is comparable to the fastest available binary descriptors such as BRIEF and ORB. In this work, we propose and experiment an original solution to 3D face recognition that supports face matching also in the case of probe scans with missing parts. In the proposed approach, distinguishing traits of the face are captured by first extracting 3D keypoints of the scan and then measuring how the face surface changes in the keypoints neighborhood using local shape descriptors. In particular: 3D keypoints detection relies on the adaptation to the case of 3D faces of the meshDOG algorithm that has been demonstrated to be effective for 3D keypoints extraction from generic objects; as 3D local descriptors we used the HOG descriptor and also proposed two alternative solutions that develop, respectively, on the histogram of orientations and the geometric histogram descriptors. Face similarity is evaluated by comparing local shape descriptors across inlier pairs of matching keypoints between probe and gallery scans. The face recognition accuracy of the approach has been first experimented on the difficult probes included in the new 2D 3D Florence face dataset that has been recently collected and released at the University of Firenze, and on the Binghamton University 3D facial expression dataset. Then, a comprehensive comparative evaluation has been performed on the Bosphorus, Gavab and UND FRGC v2.0 databases, where competitive results with respect to existing solutions for 3D face biometrics have been obtained. Graphical abstractDisplay Omitted Highlights3D face recognition approach deployable in real non-cooperative contexts of use.Fully-3D approach, based on keypoints detection, description and matching.MeshDOG keypoints detector combined with the multi-ring GH descriptor.RANSAC algorithm included for outlier removal from matching keypoints.State of the art accuracy for recognizing 3D scans with missing parts.