Cross-modality image-to-point cloud registration can be used to alleviate the aforementioned problems from the same modality registration methods. Specifically, a 3D point cloud-based map can be acquired once with Lidars, and then pose estimation can be deployed with images taken from cameras that are relatively low-maintenance and less costly on a large fleet of robots and mobile devices. Moreover, maps acquired directly with Lidars circumvents the hassle of SfM, and are largely invariant to seasonal/illumination changes. Despite the advantages of cross-modality image-to-point cloud registration, few research has been done due to its inherent difficulty. To the best of our knowledge, 2D3D-MatchNet [1]} is the only prior work on general image-to-point cloud registration. This work does cross-modal registration by learning to match image-based SIFT [2]} to point cloud-based ISS [3]} keypoints using deep metric-learning. However, the method suffers low inlier rate due to the drastic dissimilarity in the SIFT and ISS features across two modalities.
<FIGURE>